Dendrite is a second-generation Matrix homeserver written in Go!

Overview

Dendrite Build Status Dendrite Dendrite Dev

Dendrite is a second-generation Matrix homeserver written in Go. It intends to provide an efficient, reliable and scalable alternative to Synapse:

  • Efficient: A small memory footprint with better baseline performance than an out-of-the-box Synapse.
  • Reliable: Implements the Matrix specification as written, using the same test suite as Synapse as well as a brand new Go test suite.
  • Scalable: can run on multiple machines and eventually scale to massive homeserver deployments.

As of October 2020, Dendrite has now entered beta which means:

  • Dendrite is ready for early adopters. We recommend running in Monolith mode with a PostgreSQL database.
  • Dendrite has periodic semver releases. We intend to release new versions as we land significant features.
  • Dendrite supports database schema upgrades between releases. This means you should never lose your messages when upgrading Dendrite.
  • Breaking changes will not occur on minor releases. This means you can safely upgrade Dendrite without modifying your database or config file.

This does not mean:

  • Dendrite is bug-free. It has not yet been battle-tested in the real world and so will be error prone initially.
  • All of the CS/Federation APIs are implemented. We are tracking progress via a script called 'Are We Synapse Yet?'. In particular, presence and push notifications are entirely missing from Dendrite. See CHANGES.md for updates.
  • Dendrite is ready for massive homeserver deployments. You cannot shard each microservice, only run each one on a different machine.

Currently, we expect Dendrite to function well for small (10s/100s of users) homeserver deployments as well as P2P Matrix nodes in-browser or on mobile devices. In the future, we will be able to scale up to gigantic servers (equivalent to matrix.org) via polylith mode.

If you have further questions, please take a look at our FAQ or join us in:

Requirements

To build Dendrite, you will need Go 1.13 or later.

For a usable federating Dendrite deployment, you will also need:

  • A domain name (or subdomain)
  • A valid TLS certificate issued by a trusted authority for that domain
  • SRV records or a well-known file pointing to your deployment

Also recommended are:

  • A PostgreSQL database engine, which will perform better than SQLite with many users and/or larger rooms
  • A reverse proxy server, such as nginx, configured like this sample

The Federation Tester can be used to verify your deployment.

Get started

If you wish to build a fully-federating Dendrite instance, see INSTALL.md. For running in Docker, see build/docker.

The following instructions are enough to get Dendrite started as a non-federating test deployment using self-signed certificates and SQLite databases:

$ git clone https://github.com/matrix-org/dendrite
$ cd dendrite
$ ./build.sh

# Generate a Matrix signing key for federation (required)
$ ./bin/generate-keys --private-key matrix_key.pem

# Generate a self-signed certificate (optional, but a valid TLS certificate is normally
# needed for Matrix federation/clients to work properly!)
$ ./bin/generate-keys --tls-cert server.crt --tls-key server.key

# Copy and modify the config file - you'll need to set a server name and paths to the keys
# at the very least, along with setting up the database connection strings.
$ cp dendrite-config.yaml dendrite.yaml

# Build and run the server:
$ ./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml

Then point your favourite Matrix client at http://localhost:8008 or https://localhost:8448.

Progress

We use a script called Are We Synapse Yet which checks Sytest compliance rates. Sytest is a black-box homeserver test rig with around 900 tests. The script works out how many of these tests are passing on Dendrite and it updates with CI. As of November 2020 we're at around 58% CS API coverage and 83% Federation coverage, though check CI for the latest numbers. In practice, this means you can communicate locally and via federation with Synapse servers such as matrix.org reasonably well. There's a long list of features that are not implemented, notably:

  • Push
  • Search and Context
  • User Directory
  • Presence
  • Guests

We are prioritising features that will benefit single-user homeservers first (e.g Receipts, E2E) rather than features that massive deployments may be interested in (User Directory, OpenID, Guests, Admin APIs, AS API). This means Dendrite supports amongst others:

  • Core room functionality (creating rooms, invites, auth rules)
  • Federation in rooms v1-v6
  • Backfilling locally and via federation
  • Accounts, Profiles and Devices
  • Published room lists
  • Typing
  • Media APIs
  • Redaction
  • Tagging
  • E2E keys and device lists
  • Receipts

Contributing

We would be grateful for any help on issues marked as Are We Synapse Yet. These issues all have related Sytests which need to pass in order for the issue to be closed. Once you've written your code, you can quickly run Sytest to ensure that the test names are now passing.

For example, if the test Local device key changes get to remote servers was marked as failing, find the test file (e.g via grep or via the CI log output it's tests/50federation/40devicelists.pl ) then to run Sytest:

docker run --rm --name sytest
-v "/Users/kegan/github/sytest:/sytest"
-v "/Users/kegan/github/dendrite:/src"
-v "/Users/kegan/logs:/logs"
-v "/Users/kegan/go/:/gopath"
-e "POSTGRES=1" -e "DENDRITE_TRACE_HTTP=1"
matrixdotorg/sytest-dendrite:latest tests/50federation/40devicelists.pl

See sytest.md for the full description of these flags.

You can try running sytest outside of docker for faster runs, but the dependencies can be temperamental and we recommend using docker where possible.

cd sytest
export PERL5LIB=$HOME/lib/perl5
export PERL_MB_OPT=--install_base=$HOME
export PERL_MM_OPT=INSTALL_BASE=$HOME
./install-deps.pl

./run-tests.pl -I Dendrite::Monolith -d $PATH_TO_DENDRITE_BINARIES

Sometimes Sytest is testing the wrong thing or is flakey, so it will need to be patched. Ask on #dendrite-dev:matrix.org if you think this is the case for you and we'll be happy to help.

If you're new to the project, see CONTRIBUTING.md to get up to speed then look for Good First Issues. If you're familiar with the project, look for Help Wanted issues.

Hardware requirements

Dendrite in Monolith + SQLite works in a range of environments including iOS and in-browser via WASM.

For small homeserver installations joined on ~10s rooms on matrix.org with ~100s of users in those rooms, including some encrypted rooms:

  • Memory: uses around 100MB of RAM, with peaks at around 200MB.
  • Disk space: After a few months of usage, the database grew to around 2GB (in Monolith mode).
  • CPU: Brief spikes when processing events, typically idles at 1% CPU.

This means Dendrite should comfortably work on things like Raspberry Pis.

Issues
  • Implement Push Notifications

    Implement Push Notifications

    Added support for Push Notifications.according to Client-Server Push Notifications API, resolves #611.

    This PR adds a new "Pusher Database" based on the Pusher Model.

    Design thoughts 🤔

    1. Should Push Notification API be inside clientapi or in a separate component?
    2. Should Push Rules API be inside clientapi or in a separate component?
    3. What background service we should use to send /notify requests?

    What's left to do ❓

    • [x] Implement GET /_matrix/client/r0/pushers
    • [x] Implement POST /_matrix/client/r0/pushers/set
    • [x] Implement GET /_matrix/client/r0/notifications
    • [x] Implement push rules APIs - tracked in #481

    Tests 🔧

    tests/14account/01change-password.pl:

    • [x] Test 123 Pushers created with a different access token are deleted on password change...
    • [x] Test 124 Pushers created with a the same access token are not deleted on password change...

    tests/30rooms/60version_upgrade.pl:

    • [ ] Test 290 local user has push rules copied to upgraded room...
    • [ ] Test 291 remote user has push rules copied to upgraded room...

    tests/61push/01message-pushed.pl:

    • [x] Test 744 Test that a message is pushed...
    • [x] Test 745 Invites are pushed...
    • [x] Test 746 Rooms with names are correctly named in pushes...
    • [x] Test 747 Rooms with canonical alias are correctly named in pushed...
    • [x] Test 748 Rooms with many users are correctly pushed...
    • [x] Test 749 Don't get pushed for rooms you've muted...
    • [x] Test 750 Rejected events are not pushed...

    tests/61push/02add_rules.pl:

    • [x] Test 751 Can add global push rule for room...
    • [x] Test 752 Can add global push rule for sender...
    • [x] Test 753 Can add global push rule for content...
    • [x] Test 754 Can add global push rule for override...
    • [x] Test 755 Can add global push rule for underride...
    • [x] Test 756 Can add global push rule for content...
    • [x] Test 757 New rules appear before old rules by default...
    • [x] Test 758 Can add global push rule before an existing rule...
    • [x] Test 759 Can add global push rule after an existing rule...
    • [x] Test 760 Can delete a push rule...
    • [x] Test 761 Can disable a push rule...
    • [x] Test 762 Adding the same push rule twice is idempotent...

    tests/61push/03_unread_count.pl:

    • [x] Test 763 Messages that notify from another user increment notification_count...
    • [ ] Test 764 Messages that org.matrix.msc2625.mark_unread from another user increment org.matrix.msc2625.unread_count...
    • [x] Test 765 Messages that highlight from another user increment unread highlight count...

    tests/61push/05_set_actions.pl:

    • [x] Test 766 Can change the actions of default rules...
    • [x] Test 767 Changing the actions of an unknown default rule fails with 404..
    • [x] Test 768 Can change the actions of a user specified rule...
    • [x] Test 769 Changing the actions of an unknown rule fails with 404...

    tests/61push/06_get_pusher.pl:

    • [x] Test 770 Can fetch a user's pushers...

    tests/61push/06_push_rules_in_sync.pl:

    • [x] Test 771 Push rules come down in an initial /sync...
    • [x] Test 772 Adding a push rule wakes up an incremental /sync...
    • [x] Test 773 Disabling a push rule wakes up an incremental /sync...
    • [x] Test 774 Enabling a push rule wakes up an incremental /sync...
    • [x] Test 775 Setting actions for a push rule wakes up an incremental /sync...

    tests/61push/07_set_enabled.pl:

    • [x] Test 776 Can enable/disable default rules...
    • [x] Test 777 Enabling an unknown default rule fails with 404...

    tests/61push/08_rejected_pushers.pl:

    • [x] Test 778 Test that rejected pushers are removed....

    tests/61push/09_notifications_api.pl:

    • [x] Test 779 Notifications can be viewed with GET /notifications...

    tests/61push/80torture.pl:

    • [x] Test 780 Trying to add push rule with no scope fails with 400...
    • [x] Test 781 Trying to add push rule with invalid scope fails with 400...
    • [x] Test 782 Trying to add push rule with missing template fails with 400...
    • [x] Test 783 Trying to add push rule with missing rule_id fails with 400...
    • [x] Test 784 Trying to add push rule with empty rule_id fails with 400...
    • [x] Test 785 Trying to add push rule with invalid template fails with 400...
    • [x] Test 786 Trying to add push rule with rule_id with slashes fails with 400...
    • [x] Test 787 Trying to add push rule with override rule without conditions fails with 400...
    • [x] Test 788 Trying to add push rule with underride rule without conditions fails with 400...
    • [x] Test 789 Trying to add push rule with condition without kind fails with 400...
    • [x] Test 790 Trying to add push rule with content rule without pattern fails with 400...
    • [x] Test 791 Trying to add push rule with no actions fails with 400...
    • [x] Test 792 Trying to add push rule with invalid action fails with 400...
    • [x] Test 793 Trying to add push rule with invalid attr fails with 400...
    • [x] Test 794 Trying to add push rule with invalid value for enabled fails with 400...
    • [x] Test 795 Trying to get push rules with no trailing slash fails with 400...
    • [x] Test 796 Trying to get push rules with scope without trailing slash fails with 400...
    • [x] Test 797 Trying to get push rules with template without tailing slash fails with 400...
    • [x] Test 798 Trying to get push rules with unknown scope fails with 400...
    • [x] Test 799 Trying to get push rules with unknown template fails with 400...
    • [x] Test 800 Trying to get push rules with unknown attribute fails with 400...
    • [x] Test 801 Trying to get push rules with unknown rule_id fails with 404...

    Pull Request Checklist ✅

    • [x] I have added any new tests that need to pass to sytest-whitelist as specified in docs/sytest.md
    • [x] Pull request includes a sign off

    Signed-off-by: Dan Peleg <[email protected]>

    opened by danpe 65
  • end-to-end serverside implementation

    end-to-end serverside implementation

    1. /query
    2. /claim
    3. /upLoad
    4. /change and extension dock interface through kafka

    in order to keep fashioned to the latest implementation and what may come in the future , I add e2e module in a single server as encryptoapi , then it's painless to change implementation .

    stale 
    opened by fadeAce 37
  • Implements room tagging.

    Implements room tagging.

    opened by SUMUKHA-PK 32
  • Unable to join room due to invalid signature verification

    Unable to join room due to invalid signature verification

    Problems with the certificate of the home server of one of the participants in the room, does not allow you to join the room!

    ERROR listing:

    time="2020-09-10T11:15:21.069528376Z" level=error msg="Failed to fetch key for server" func="github.com/matrix-org/gomatrixserverlib.(*DirectKeyFetcher).FetchKeys.func1" file="/root/go/pkg/mod/github.com/matrix-org/[email protected]/keyring.go:482" context=missing error="Get matrix://netwhood.online/_matrix/key/v2/server: x509: certificate is valid for matrix.netwhood.online, not netwhood.online" fetcher=DirectKeyFetcher

    bug 
    opened by llacqie 29
  • Monolithic server client API becomes unresponsive until restarted

    Monolithic server client API becomes unresponsive until restarted

    Background information

    • Dendrite version or git SHA: bca2790c678887232266c5726be79b80ddd9930b (post-dev v0.3.4 for https://github.com/matrix-org/dendrite/pull/1672)
    • Monolith or Polylith?: Monolith
    • SQLite3 or Postgres?: Postgres
    • Running in Docker?: Docker
    • go version: official images

    Description

    My Dendrite server has become unresponsive two times in a 24 hour period. This is noticed by Element web (app.element.io) saying the server is offline. The console logs show sync endpoints timing out.

    Restarting the docker container has caused the issue to resolve on both times.

    Nothing interesting found in logs that I could spot, errors mainly related to fetching of remote auth events. Will submit a log file privately that covers the second time with an issue.

    Dendrite is v0.3.4 post dev on hash bca2790c678887232266c5726be79b80ddd9930b (which is a part of https://github.com/matrix-org/dendrite/pull/1672). MSC2836 is enabled in settings.

    While the unresponsiveness continues, the logs do show activity in the federation API's.

    bug 
    opened by jaywink 22
  • State migration fails while rewriting snapshots after upgrading to v0.4.0

    State migration fails while rewriting snapshots after upgrading to v0.4.0

    Background information

    • Dendrite version or git SHA: v0.4.0
    • Monolith or Polylith?: Monolith
    • SQLite3 or Postgres?: Postgres
    • Running in Docker?: No
    • go version: 1.13.8

    Description

    I'm not sure how to interpret this error. Clearly the state storage upgrade did not complete, despite the log saying it did.

    Jul 15 22:54:01 dendrite-host dendrite-monolith-server[169231]: time="2021-07-16T03:54:01.897195913Z" level=warning msg="Rewriting snapshots 3000-3100 of 21375..." func="UpStateBlocksRefactor\n\t" file=" [2021041615092700_state_blocks_refactor.go:140]"
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: time="2021-07-16T03:54:02.667781275Z" level=warning msg="State storage upgrade complete" func="UpStateBlocksRefactor\n\t" file=" [2021041615092700_state_blocks_refactor.go:178]"
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: time="2021-07-16T03:54:02.679749124Z" level=panic msg="failed to connect to room server db" func="NewInternalAPI\n\t" file=" [roomserver.go:54]" error="RunDeltas: Failed run migration: ERROR 2021041615092700_state_blocks_refactor.go: failed to run Go migration function func(*sql.Tx) error: cannot xref null state block with snapshot 13936: sql: no rows in result set"
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: panic: (*logrus.Entry) (0x1261380,0xc00011e230)
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: goroutine 1 [running]:
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: github.com/sirupsen/logrus.(*Entry).log(0xc00011e1c0, 0x0, 0xc000198240, 0x23)
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]:         github.com/sirupsen/[email protected]/entry.go:259 +0x2e2
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: github.com/sirupsen/logrus.(*Entry).Log(0xc00011e1c0, 0x0, 0xc000271b88, 0x1, 0x1)
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]:         github.com/sirupsen/[email protected]/entry.go:285 +0x86
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: github.com/sirupsen/logrus.(*Entry).Logf(0xc00011e1c0, 0x0, 0x1297d7d, 0x23, 0x0, 0x0, 0x0)
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]:         github.com/sirupsen/[email protected]/entry.go:330 +0xe2
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: github.com/sirupsen/logrus.(*Entry).Panicf(...)
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]:         github.com/sirupsen/[email protected]/entry.go:368
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: github.com/matrix-org/dendrite/roomserver.NewInternalAPI(0xc00011c630, 0x146f860, 0xc0002a1cb0, 0xc0000c7580, 0x149ad00)
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]:         github.com/matrix-org/[email protected]/roomserver/roomserver.go:54 +0x237
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]: main.main()
    Jul 15 22:54:02 dendrite-host dendrite-monolith-server[169231]:         github.com/matrix-org/[email protected]/cmd/dendrite-monolith-server/main.go:86 +0x27c
    Jul 15 22:54:02 dendrite-host systemd[1]: dendrite.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
    Jul 15 22:54:02 dendrite-host systemd[1]: dendrite.service: Failed with result 'exit-code'.
    

    Steps to reproduce

    • Upgrade Dendrite to v0.4.0
    • Dendrite attempts state migration on start
    needs investigation 
    opened by jellykells 21
  • storage: state NIDs missing from the database (0 != 1)

    storage: state NIDs missing from the database (0 != 1)"

    Background information

    • Dendrite version or git SHA: Dendrite version 0.3.11+464b908b
    • Monolith or Polylith?: mono
    • SQLite3 or Postgres?: psql
    • Running in Docker?: no f way
    • go version: go version go1.14 linux/amd64

    Description

    After git pull nothing works anymore, POSTs get 500 internal server errors

    ERRO[2021-04-26T21:14:14.000494976Z] [github.com/matrix-org/dendrite/clientapi/routing/membership.go:385] checkMemberInRoom
             QueryCurrentState: could not query membership for user  error="storage: state NIDs missing from the database (0 != 1)" req.id=dlCjrEVVp0pJ req.method=POST req.path="/_matrix/client/r0/rooms/!yomrOFwgFXzmeMAbzX:matrix.org/read_markers" user_id="@grin:dendrite.grin.hu"
    
    bug component:roomserver 
    opened by grinapo 21
  • Handle trailing slashes at the end of paths

    Handle trailing slashes at the end of paths

    The HTTP routers Dendrite uses see /foo/bar and /foo/bar/ as two different routes. We'd ideally not want that, because Synapse and sytest sometimes add trailing slashes at the end of paths even though the specs doesn't use any.

    This might require a bit of research but should be quite straightforward.

    maintenance good first issue 
    opened by babolivier 19
  • stuck in a #dendrite:matrix.org related loop, heavy resource usage

    stuck in a #dendrite:matrix.org related loop, heavy resource usage

    Background information

    • Dendrite version or git SHA: 0.6.3 and 5106cc807cf22a95420b24f6bfdd5c9ac8aa06de
    • Monolith or Polylith?: monolith
    • SQLite3 or Postgres?: postgres
    • Running in Docker?: no
    • go version: 1.17.6
    • Client used (if applicable): element, curl

    Description

    • What is the problem: dendrite is stuck in a loop printing the same errors
    • Who is affected:
    • How is this bug manifesting: steadily increasing ram and jetstream storage size
    • When did this first appear: some time after updating to 0.6.3

    Steps to reproduce

    level=error msg="syncapi: failed to QuerySharedUsers for key change event from key server" error="found 2577 users but only have state key nids for 2350 of them"
    level=error msg="syncapi: failed to QuerySharedUsers for key change event from key server" error="found 2448 users but only have state key nids for 2221 of them"
    
    • it has something to do with #dendrite:matrix.org because that's the only room with that many users that this server has participated in
    • these print at about 8 lines per second continuously
    • restarting and deleting ~/jetstream have no effect
    • ~/jetstream is growing steadily
    • joining and leaving #dendrite:matrix.org has no effect
    • forgetting the room has no effect #2176
    bug 
    opened by bones-was-here 15
  • Crashes When Joining Large Room (Memory Usage Loading History)

    Crashes When Joining Large Room (Memory Usage Loading History)

    Background information

    • Dendrite version or git SHA: both the latest commit (from github) and the latest release (0.4.0)
    • Monolith or Polylith?: monolith
    • SQLite3 or Postgres?: SQLite3
    • Running in Docker?: no
    • go version: go version go1.15.9 linux/amd64

    Description

    I'm using the latest commit as of writing this. (I have experienced the same issue with the latest "release" as well.)

    When I launch my server, it seems to work well until I try to join a room on another server with too much history.

    Whenever I join any room on another server with a lot of messages, the process of joining the room hangs (visibly, seemingly forever) until my memory usage climbs so high that I have to power off my server (because there's no longer enough memory for me to spawn a shell and kill the process).

    Edit: Sometimes, but not always, after it reaches this point, my scheduler appears to kill the dendrite server and I don't have to reboot.

    Steps to reproduce

    I'm not doing anything too complicated:

    cd dendrite
    ./bin/dendrite-monolith-server --tls-cert /etc/letsencrypt/live/my_site.com/fullchain.pem --tls-key /etc/letsencrypt/live/my_site.com/privkey.pem --config dendrite.yaml
    

    Here is my dendrite.yaml file.

    The console output when I attempt to join other servers is just a seemingly-never-ending stream of INFO statements like the following:

    INFO[2021-07-26T00:00:48.443092992Z] [send.go:206] Send
       Received transaction "1627243433681" from "techlore.net" containing 0 PDUs, 1 EDUs  req.id=7qSuTVlA8RCg req.method=PUT req.path=/_matrix/federation/v1/send/1627243433681
    INFO[2021-07-26T00:00:49.138694805Z] [send.go:206] Send
       Received transaction "1627243433905" from "techlore.net" containing 0 PDUs, 1 EDUs  req.id=AKG6OVWhIwT3 req.method=PUT req.path=/_matrix/federation/v1/send/1627243433905
    

    I think dendrite is trying to load all history since the beginning of time for every room on every server I join. It just keeps receiving transactions for messages until I run out of memory. One would expect that by default it would only load maybe a couple hundred messages or so, or at least that it would decide to stop loading history after it eats all available memory.

    bug component:federation 
    opened by SentToDevNull 15
  • Support other database backends

    Support other database backends

    It would be great if other database backends, such as SQLite or ql were supported too. I'd rather not have to install postgresql and configure it just to run a homeserver, it seems like a lot of extra hassle.

    opened by Ronsor 15
  • Try to reconstuct membership events if it's a federated invite

    Try to reconstuct membership events if it's a federated invite

    This should fix https://github.com/matrix-org/dendrite/issues/2121

    The issue here is that /rooms/{roomID}/members was failing with "You aren't a member of the room and weren't previously a member of the room.", as we don't have a membershipNID for invites coming over federation, and we don't know anything about the room yet. This tries to reconstruct the membership events for invites coming in over federation, making Element Web a bit more happy.

    component:roomserver 
    opened by S7evinK 3
  • Retire all pending invites for a given user and room

    Retire all pending invites for a given user and room

    If you receive multiple distinct invites for the same room, it becomes impossible to retire them all because the user is no longer invited to the room but there are still pending invites in the sync API.

    This modifies the shape so that the sync API cleans up all pending invites for a given user for a given room instead, so that users shouldn't get stuck with invites they reject but remain stuck with.

    component:roomserver component:syncapi 
    opened by neilalexander 0
  • How to debug push notification issues ?

    How to debug push notification issues ?

    Background information

    • Dendrite version or git SHA: 0.9.1
    • Monolith or Polylith?: monolyth
    • SQLite3 or Postgres?: postgres
    • Running in Docker?: no
    • go version: 1.18
    • Client used (if applicable): element

    Description

    • What is the problem: when debugging notifications with element, element is waiting for the push notification that doesn't come
    • Who is affected: me
    • How is this bug manifesting: I have no push notification on element
    • When did this first appear: dendrite 0.8.9 I think, before that it worked well

    Steps to reproduce

    • open element
    • debug notification
    • element get google fcm token etc., every tick is green
    • wait for "push notification" from dendrite

    There isn't much logs related to this issue.

    time="2022-08-12T13:59:59.810507178Z" level=info msg="Cleaning old notifications" func=github.com/matrix-org/dendrite/userapi.NewInternalAPI.func1 file="/opt/dendrite/userapi/userapi.go:101"
    time="2022-08-12T14:00:10.311907582Z" level=debug msg="Responding to sync since client gave up or timeout was reached" func="github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).OnIncomingSyncRequest.func1" file="/opt/dendrite/syncapi/sync/requestpool.go:268" device_id=1Ih6kxxx limit=20 req.id=6Xp2MXvqxxxl req.method=GET req.path=/_matrix/client/r0/sync since=s568244_0_197_610_567943_568246_55_107770_0 timeout=30s user_id="@user:dendrite.server.example"
    time="2022-08-12T14:00:23.227203919Z" level=debug msg="Database returned 1 pushers" func="github.com/matrix-org/dendrite/userapi/storage/postgres.(*pushersStatements).SelectPushers" file="/opt/dendrite/userapi/storage/postgres/pusher_table.go:139"
    time="2022-08-12T14:00:40.529283666Z" level=debug msg="Responding to sync since client gave up or timeout was reached" func="github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).OnIncomingSyncRequest.func1" file="/opt/dendrite/syncapi/sync/requestpool.go:268" device_id=1Ih6kxxx limit=20 req.id=PAC15XNmxxxx req.method=GET req.path=/_matrix/client/r0/sync since=s568244_0_197_610_567943_568246_55_107770_0 timeout=30s user_id="@user:dendrite.server.example"
    

    I also found this log, just after receiving a message :

    time="2022-08-12T13:42:03.664811412Z" level=info msg=PerformPusherCreation func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).PerformPusherSet" file="/opt/dendrite/userapi/internal/api.go:699" display_name=Element localpart=marc pushkey="eTpEjS2aS2eabktg1jqSUe:APA91bxxx" req.id=C6l4Needxxx req.method=POST req.path=/_matrix/client/r0/pushers/set user_id="@user:dendrite.server.example"
    

    Is there a way to better debug issues with push notifications ? Everything seems fine, but element cannot receive push notifications.

    opened by Notmarrco 0
  • Add SQLite WAL and busy_timeout

    Add SQLite WAL and busy_timeout

    SQLite has rather unfortunate defaults for concurrent accesses, leading to database locked errors. https://github.com/matrix-org/dendrite/pull/1290 appears to try to fix some of this, but doesn't appear to cover a concurrent read+write and the errors still happened in large quantities when I joined the dendrite room.

    This PR enables WAL for concurrent read+write and much better performance. It also sets busy_timeout to 10 seconds to not immediately fail operations while the database is locked, providing a DB level solution to PR 1290's objective with additional coverage.

    I chose this particular location to add the two statements as they should apply to every SQLite DB connection.

    From a quick test it looks as if the db locked errors are gone with this change.

    Pull Request Checklist

    • [ ] I have added added tests for PR or I have justified why this PR doesn't need tests.
    • [ ] Pull request includes a sign off

    Signed-off-by: Your Name <[email protected]>

    opened by sfPlayer1 1
  • Rewrite Startup for using cobra (and koanf)

    Rewrite Startup for using cobra (and koanf)

    improvements are:

    • one binary also for service like

      • [ ] create-account
      • [x] generete-config
      • [x] generate-keys
      • [x] monolith
      • [ ] polylith (with there components)
      • [x] version
    • config reading with:

      • toml
      • yaml
      • json
      • env
        • works level of struct are __ (two underscore - so e.g. server_name is the same on yaml):
        • e.g. set DENDRITE_GLOBAL__SERVER_NAME
        • [ ] split database uri into components (e.g. for set password on another way then host - k8s secrets vs. configmaps)
    • config writing (by generete-config with env and cli-flags configurated):

      • toml
      • yaml
      • json

    fix #2591

    PS: viper has some problems with parsing env and is not possible to disable file reading


    that would be the new structure: if you like it, i will reimplement the rest - please feedback @neilalexander


    CMD Helps

    Dendrite is a second-generation Matrix homeserver written in Go.
    It intends to provide an efficient, reliable and scalable alternative to Synapse.
    
    Usage:
      dendrite [command]
    
    Available Commands:
      completion      Generate the autocompletion script for the specified shell
      generate-config generate configuration file
      generate-keys   Generate key files which are required by dendrite.
      help            Help about any command
      serve           Run dendrite server
      version         Print the version number of Dendrite
    
    Flags:
      -h, --help   help for dendrite
    
    Use "dendrite [command] --help" for more information about a command.
    
    generate-config
    generate default configuration (overwrite is possible by commandline arguments and enviroment variables
    
    Usage:
      dendrite generate-config [flags]
    
    Flags:
          --ci              sane defaults for CI testing
      -c, --config string   write config to file if not set to print in yaml
          --db string       The DB URI to use for all components if not SQLite files
      -h, --help            help for generate-config
          --server string   The domain name of the server if not 'localhost'
    
    generate-keys
    Nothing todo, please set argument
    
    Usage:
      dendrite generate-keys [flags]
    
    Flags:
      -h, --help                        help for generate-keys
          --private-key string          An Ed25519 private key to generate for use for object signing
          --server string               Optional: Create TLS certificate/keys with this domain name set. Useful for integration testing.
          --tls-authority-cert string   Optional: Create TLS certificate/keys based on this CA authority. Useful for integration testing.
          --tls-authority-key string    Optional: Create TLS certificate/keys based on this CA authority. Useful for integration testing.
          --tls-cert string             An X509 certificate file to generate for use for TLS
          --tls-key string              An RSA private key file to generate for use for TLS
    
    serve (monolith)
    Run dendrite server in monolith modus (if no component called)
    
    Usage:
      dendrite serve [flags]
      dendrite serve [command]
    
    Available Commands:
      appservice  Run dendrite server polylith component appservice
    
    Flags:
          --api                               Use HTTP APIs instead of short-circuiting (warning: exposes API endpoints!)
          --api-bind-address string           The HTTP listening port for the internal HTTP APIs (if -api is enabled) (default "localhost:18008")
      -c, --config string                     config file
      -h, --help                              help for serve
          --http-bind-address string          The HTTP listening port for the server (default ":8008")
          --https-bind-address string         The HTTPS listening port for the server (default ":8448")
          --really-enable-open-registration   This allows open registration without secondary verification (reCAPTCHA). This is NOT RECOMMENDED and will SIGNIFICANTLY increase the risk that your server will be used to send spam or conduct attacks, which may result in your server being banned from rooms.
          --tls-cert string                   The PEM formatted X509 certificate to use for TLS
          --tls-key string                    The PEM private key to use for TLS
    
    Use "dendrite serve [command] --help" for more information about a command.
    
    
    serve appservice (current state - polylith component appservice)
    Run dendrite server polylith component appservice
    
    Usage:
      dendrite serve appservice [flags]
    
    Flags:
      -h, --help   help for appservice
    
    Global Flags:
      -c, --config string                     config file
          --really-enable-open-registration   This allows open registration without secondary verification (reCAPTCHA). This is NOT RECOMMENDED and will SIGNIFICANTLY increase the risk that your server will be used to send spam or conduct attacks, which may result in your server being banned from rooms.
    
    
    version
    All software has versions. This is Dendrites
    
    Usage:
      dendrite version [flags]
    
    Flags:
      -h, --help   help for version
    
    opened by genofire 3
Releases(v0.9.3)
  • v0.9.3(Aug 15, 2022)

    Important

    • This is a security release to fix a vulnerability within event auth, affecting all versions of Dendrite before 0.9.3. Upgrading to this version is highly recommended. For more information, see here.

    Fixes

    • Dendrite will now correctly parse the "events_default" power level value for event auth.
    Source code(tar.gz)
    Source code(zip)
  • v0.9.2(Aug 12, 2022)

    Features

    • Dendrite now supports history visibility on the /sync, /messages and /context endpoints
      • It should now be possible to view the history of a room in more cases (as opposed to limiting scrollback to the join event or defaulting to the restrictive "join" visibility rule as before)
    • The default room version for newly created rooms is now room version 9
    • New admin endpoint /_dendrite/admin/resetPassword/{userID} has been added, which replaces the -reset-password flag in create-account
    • The create-account binary now uses shared secret registration over HTTP to create new accounts, which fixes a number of problems with account data and push rules not being configured correctly for new accounts
    • The internal HTTP APIs for polylith deployments have been refactored for correctness and consistency
    • The federation API will now automatically clean up some EDUs that have failed to send within a certain period of time
    • The /hierarchy endpoint will now return potentially joinable rooms (contributed by texuf)
    • The user directory will now show or hide users correctly

    Fixes

    • Send-to-device messages should no longer be incorrectly duplicated in /sync
    • The federation sender will no longer create unnecessary destination queues as a result of a logic error
    • A bug where database migrations may not execute properly when upgrading from older versions has been fixed
    • A crash when failing to update user account data has been fixed
    • A race condition when generating notification counts has been fixed
    • A race condition when setting up NATS has been fixed (contributed by brianathere)
    • Stale cache data for membership lazy-loading is now correctly invalidated when doing a complete sync
    • Data races within user-interactive authentication have been fixed (contributed by tak-hntlabs)
    Source code(tar.gz)
    Source code(zip)
  • v0.9.1(Aug 3, 2022)

    Fixes

    • Upgrades a dependency which caused issues building Dendrite with Go 1.19
    • The roomserver will no longer give up prematurely after failing to call /state_ids
    • Removes the faulty room info cache, which caused of a number of race conditions and occasional bugs (including when creating and joining rooms)
    • The media endpoint now sets the Cache-Control header correctly to prevent web-based clients from hitting media endpoints excessively
    • The sync API will now advance the PDU stream position correctly in all cases (contributed by sergekh2)
    • The sync API will now delete the correct range of send-to-device messages when advancing the stream position
    • The device list changed key in the /sync response should now return the correct users
    • A data race when looking up missing state has been fixed
    • The /send_join API is now applying stronger validation to the received membership event
    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Aug 1, 2022)

    Please note that Go 1.18 is now required to build this version.

    Features

    • Dendrite now uses Ristretto for managing in-memory caches
      • Should improve cache utilisation considerably over time by more intelligently selecting and managing cache entries compared to the previous LRU-based cache
      • Defaults to a 1GB cache size if not configured otherwise
      • The estimated cache size in memory and maximum age can now be configured with new configuration options to prevent unbounded cache growth
    • Added support for serving the /.well-known/matrix/client hint directly from Dendrite
    • Refactored membership updater, which should eliminate some bugs caused by the membership table getting out of sync with the room state
    • The User API is now responsible for sending account data updates to other components, which may fix some races and duplicate account data events
    • Optimised database query for checking whether a remote server is allowed to request an event over federation without using anywhere near as much CPU time (PostgreSQL only)
    • Database migrations have been refactored to eliminate some problems that were present with goose and upgrading from older Dendrite versions
    • Media fetching will now use the /v3 endpoints for downloading media from remote homeservers
    • HTTP 404 and HTTP 405 errors from the client-facing APIs should now be returned with CORS headers so that web-based clients do not produce incorrect access control warnings for unknown endpoints
    • Some preparation work for full history visibility support

    Fixes

    • Fixes a crash that could occur during event redaction
    • The /members endpoint will no longer incorrectly return HTTP 500 as a result of some invite events
    • Send-to-device messages should now be ordered more reliably and the last position in the stream updated correctly
    • Parsing of appservice configuration files is now less strict (contributed by Kab1r)
    • The sync API should now identify shared users correctly when waking up for E2EE key changes
    • The federation /state endpoint will now return a HTTP 403 when the state before an event isn't known instead of a HTTP 500
    • Presence timestamps should now be calculated with the correct precision
    • A race condition in the roomserver's room info has been fixed
    • A race condition in the sync API has been fixed
    Source code(tar.gz)
    Source code(zip)
  • v0.8.9(Jul 1, 2022)

    Features

    • Incoming device list updates over federation are now queued in JetStream for processing so that they will no longer block incoming federation transactions and should never end up dropped, which will hopefully help E2EE reliability
    • The /context endpoint now returns "start" and "end" parameters to allow pagination from a context call
    • The /messages endpoint will no longer return "end" when there are no more messages remaining
    • Deactivated user accounts will now leave all rooms automatically
    • New admin endpoint /_dendrite/admin/evacuateUser/{userID} has been added for forcing a local user to leave all joined rooms
    • Dendrite will now automatically attempt to raise the file descriptor limit at startup if it is too low

    Fixes

    • A rare crash when retrieving remote device lists has been fixed
    • Fixes a bug where events were not redacted properly over federation
    • The /invite endpoints will now return an error instead of silently proceeding if the user ID is obviously malformed
    Source code(tar.gz)
    Source code(zip)
  • v0.8.8(Jun 9, 2022)

    Features

    • The performance of state resolution has been increased significantly for larger rooms
    • A number of changes have been made to rate limiting:
      • Logged in users will now be rate-limited on a per-session basis rather than by remote IP
      • Rate limiting no longer applies to admin or appservice users
      • It is now possible to configure additional users that are exempt from rate limiting using the exempt_user_ids option in the rate_limiting section of the Dendrite config
    • Setting state is now idempotent via the client API state endpoints

    Fixes

    • Room upgrades now properly propagate tombstone events to remote servers
    • Room upgrades will no longer send tombstone events if creating the upgraded room fails
    • A crash has been fixed when evaluating restricted room joins
    Source code(tar.gz)
    Source code(zip)
  • v0.8.7(Jun 1, 2022)

    Features

    • Support added for room version 10

    Fixes

    • A number of state handling bugs have been fixed, which previously resulted in missing state events, unexpected state deletions, reverted memberships and unexpectedly rejected/soft-failed events in some specific cases
    • Fixed destination queue performance issues as a result of missing indexes, which speeds up outbound federation considerably
    • A bug which could cause the /register endpoint to return HTTP 500 has been fixed
    Source code(tar.gz)
    Source code(zip)
  • v0.8.6(May 26, 2022)

    Features

    • Room versions 8 and 9 are now marked as stable
    • Dendrite can now assist remote users to join restricted rooms via /make_join and /send_join

    Fixes

    • The sync API no longer returns immediately on /sync requests unnecessarily if it can be avoided
    • A race condition has been fixed in the sync API when updating presence via /sync
    • A race condition has been fixed sending E2EE keys to remote servers over federation when joining rooms
    • The trusted_private_chat preset should now grant power level 100 to all participant users, which should improve the user experience of direct messages
    • Invited users are now authed correctly in restricted rooms
    • The join_authorised_by_users_server key is now correctly stripped in restricted rooms when updating the membership event
    • Appservices should now receive invite events correctly
    • Device list updates should no longer contain optional fields with null values
    • The /deactivate endpoint has been fixed to no longer confuse Element with incorrect completed flows
    Source code(tar.gz)
    Source code(zip)
  • v0.8.5(May 13, 2022)

    Features

    Fixes

    • Monolith deployments will no longer panic at startup if given a config file that does not include the internal_api and external_api options
    • State resolution v2 now correctly identifies other events related to power events, which should fix some event auth issues
    • The latest events updater will no longer implicitly trust the new forward extremities when calculating the current room state, which may help to avoid some state resets
    • The one-time key count is now correctly returned in /sync even if the request otherwise timed out, which should reduce the chance that unnecessary one-time keys will be uploaded by clients
    • The create-account tool should now work properly when the database is configured using the global connection pool
    Source code(tar.gz)
    Source code(zip)
  • v0.8.4(May 10, 2022)

    Fixes

    • Fixes a regression introduced in the previous version where appservices, push and phone-home statistics would not work over plain HTTP
    • Adds missing indexes to the sync API output events table, which should significantly improve /sync performance and reduce database CPU usage
    • Building Dendrite with the bimg thumbnailer should now work again (contributed by database64128)
    Source code(tar.gz)
    Source code(zip)
  • v0.8.3(May 9, 2022)

    Features

    • Open registration is now harder to enable, which should reduce the chance that Dendrite servers will be used to conduct spam or abuse attacks
      • Dendrite will only enable open registration if you pass the --really-enable-open-registration command line flag at startup
      • If open registration is enabled but this command line flag is not passed, Dendrite will fail to start up
    • Dendrite now supports phone-home statistic reporting
      • These statistics include things like the number of registered and active users, some configuration options and platform/environment details, to help us to understand how Dendrite is used
      • This is not enabled by default — it must be enabled in the global.report_stats section of the config file
    • Monolith installations can now be configured with a single global database connection pool (in global.database in the config) rather than having to configure each component separately
      • This also means that you no longer need to balance connection counts between different components, as they will share the same larger pool
      • Specific components can override the global database settings by specifying their own database block
      • To use only the global pool, you must configure global.database and then remove the database block from all of the component sections of the config file
    • A new admin API endpoint /_dendrite/admin/evacuateRoom/{roomID} has been added, allowing server admins to forcefully part all local users from a given room
    • The sync notifier now only loads members for the relevant rooms, which should reduce CPU usage and load on the database
    • A number of component interfaces have been refactored for cleanliness and developer ease
    • Event auth errors in the log should now be much more useful, including the reason for the event failures
    • The forward extremity calculation in the roomserver has been simplified
    • A new index has been added to the one-time keys table in the keyserver which should speed up key count lookups

    Fixes

    • Dendrite will no longer process events for rooms where there are no local users joined, which should help to reduce CPU and RAM usage
    • A bug has been fixed in event auth when changing the user levels in m.room.power_levels events
    • Usernames should no longer be duplicated when no room name is set
    • Device display names should now be correctly propagated over federation
    • A panic when uploading cross-signing signatures has been fixed
    • Presence is now correctly limited in /sync based on the filters
    • The presence stream position returned by /sync will now be correct if no presence events were returned
    • The media /config endpoint will no longer return a maximum upload size field if it is configured to be unlimited in the Dendrite config
    • The server notices room will no longer produce "User is already joined to the room" errors
    • Consumer errors will no longer flood the logs during a graceful shutdown
    • Sync API and federation API consumers will no longer unnecessarily query added state events matching the one in the output event
    • The Sync API will no longer unnecessarily track invites for remote users
    Source code(tar.gz)
    Source code(zip)
  • v0.8.2(Apr 27, 2022)

    Features

    • Lazy-loading has been added to the /sync endpoint, which should speed up syncs considerably
    • Filtering has been added to the /messages endpoint
    • The room summary now contains "heroes" (up to 5 users in the room) for clients to display when no room name is set
    • The existing lazy-loading caches will now be used by /messages and /context so that member events will not be sent to clients more times than necessary
    • The account data stream now uses the provided filters
    • The built-in NATS Server has been updated to version 2.8.0
    • The /state and /state_ids endpoints will now return M_NOT_FOUND for rejected events
    • Repeated calls to the /redact endpoint will now be idempotent when a transaction ID is given
    • Dendrite should now be able to run as a Windows service under Service Control Manager

    Fixes

    • Fictitious presence updates will no longer be created for users which have not sent us presence updates, which should speed up complete syncs considerably
    • Uploading cross-signing device signatures should now be more reliable, fixing a number of bugs with cross-signing
    • All account data should now be sent properly on a complete sync, which should eliminate problems with client settings or key backups appearing to be missing
    • Account data will now be limited correctly on incremental syncs, returning the stream position of the most recent update rather than the latest stream position
    • Account data will not be sent for parted rooms, which should reduce the number of left/forgotten rooms reappearing in clients as empty rooms
    • The TURN username hash has been fixed which should help to resolve some problems when using TURN for voice calls (contributed by fcwoknhenuxdfiyv)
    • Push rules can no longer be modified using the account data endpoints
    • Querying account availability should now work properly in polylith deployments
    • A number of bugs with sync filters have been fixed
    • A default sync filter will now be used if the request contains a filter ID that does not exist
    • The pushkey_ts field is now using seconds instead of milliseconds
    • A race condition when gracefully shutting down has been fixed, so JetStream should no longer cause the process to exit before other Dendrite components are finished shutting down
    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Apr 7, 2022)

  • v0.8.0(Apr 7, 2022)

    Features

    • Support for presence has been added
      • Presence is not enabled by default
      • The global.presence.enable_inbound and global.presence.enable_outbound configuration options allow configuring inbound and outbound presence separately
    • Support for room upgrades via the /room/{roomID}/upgrade endpoint has been added (contributed by DavidSpenler, alexkursell)
    • Support for ignoring users has been added
    • Joined and invite user counts are now sent in the /sync room summaries
    • Queued federation and stale device list updates will now be staggered at startup over an up-to 2 minute warm-up period, rather than happening all at once
    • Memory pressure created by the sync notifier has been reduced
    • The EDU server component has now been removed, with the work being moved to more relevant components

    Fixes

    • It is now possible to set the power_level_content_override when creating a room to include power levels over 100
    • /send_join and /state responses will now not unmarshal the JSON twice
    • The stream event consumer for push notifications will no longer request membership events that are irrelevant
    • Appservices will no longer incorrectly receive state events twice
    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Mar 25, 2022)

    Features

    • The roomserver input API will now queue all events into NATS, which provides better crash resilience
    • The roomserver input API now configures per-room consumers, which should use less memory
    • Canonical aliases can now be added and removed
    • MSC2946 Spaces Summary now works correctly, both locally and over federation
    • Healthcheck endpoints are now available at:
      • /_dendrite/monitor/up, which will return 200 when Dendrite is ready to accept requests
      • /_dendrite/monitor/health, which will return 200 if healthy and 503 if degraded for some reason
    • The X-Matrix federation authorisation header now includes a destination field, as per MSC3383
    • The /sync endpoint now uses less memory by only ranging state for rooms that the user has participated in
    • The /messages endpoint now accepts stream positions in both the from and to parameters
    • Dendrite will now log a warning at startup if the file descriptor limit is set too low
    • The federation client will now attempt to use HTTP/2 if available
    • The federation client will now attempt to resume TLS sessions if possible, to reduce handshake overheads
    • The built-in NATS Server has been updated to version 2.7.4
    • NATS streams that don't match the desired configuration will now be recreated automatically
    • When performing a graceful shutdown, Dendrite will now wait for NATS Server to shutdown completely, which should avoid some corruption of data on-disk
    • The create-account tool has seen a number of improvements, will now ask for passwords automatically

    Fixes

    • The /sync endpoint will no longer lose state events when truncating the timeline for history visibility
    • The /context endpoint now works correctly with lazy_load_members
    • The /directory/list/room/{roomID} endpoint now correctly reports whether a room is published in the server room directory or not
    • Some bugs around appservice username validation have been fixed
    • Roomserver output messages are no longer unnecessarily inflated by state events, which should reduce the number of NATS message size errors
    • Stream IDs for device list updates are now always 64-bit, which should fix some problems when running Dendrite on a 32-bit system
    • Purging room state in the sync API has been fixed after a faulty database query was corrected
    • The federation client will now release host records for remote destinations after 5 minutes instead of holding them in memory forever
    • Remote media requests will now correctly return an error if the file cannot be found or downloaded
    • A panic in the media API that could happen when the remote file doesn't exist has been fixed
    • Various bugs around membership state and invites have been fixed
    • The memberships table will now be correctly updated when rejecting a federated invite
    • The client API and appservice API will now access the user database using the user API rather than accessing the database directly
    Source code(tar.gz)
    Source code(zip)
  • v0.6.5(Mar 4, 2022)

    Features

    • Early support for push notifications has been added, with support for push rules, pushers, HTTP push gateways and the /notifications endpoint (contributions by danpe, PiotrKozimor and tommie)
    • Spaces Summary (MSC2946) is now correctly supported (when msc2946 is enabled in the config)
    • All media API endpoints are now available under the /v3 namespace
    • Profile updates (display name and avatar) are now sent asynchronously so they shouldn't block the client for a very long time
    • State resolution v2 has been optimised further to considerably reduce the number of memory allocations
    • State resolution v2 will no longer duplicate events unnecessarily when calculating the auth difference
    • The create-account tool now has a -reset-password option for resetting the passwords of existing accounts
    • The /sync endpoint now calculates device list changes much more quickly with less RAM used
    • The /messages endpoint now lazy-loads members correctly

    Fixes

    • Read receipts now work correctly by correcting bugs in the stream positions and receipt coalescing
    • Topological sorting of state and join responses has been corrected, which should help to reduce the number of auth problems when joining new federated rooms
    • Media thumbnails should now work properly after having unnecessarily strict rate limiting removed
    • The roomserver no longer holds transactions for as long when processing input events
    • Uploading device keys and cross-signing keys will now correctly no-op if there were no changes
    • Parameters are now remembered correctly during registration
    • Devices can now only be deleted within the appropriate UIA flow
    • The /context endpoint now returns 404 instead of 500 if the event was not found
    • SQLite mode will no longer leak memory as a result of not closing prepared statements
    Source code(tar.gz)
    Source code(zip)
  • v0.6.4(Feb 22, 2022)

    Features

    • All Client-Server API endpoints are now available under the /v3 namespace
    • The /whoami response format now matches the latest Matrix spec version
    • Support added for the /context endpoint, which should help clients to render quote-replies correctly
    • Accounts now have an optional account type field, allowing admin accounts to be created
    • Server notices are now supported
    • Refactored the user API storage to deduplicate a significant amount of code, as well as merging both user API databases into a single database
      • The account database is now used for all user API storage and the device database is now obsolete
      • For some installations that have separate account and device databases, this may result in access tokens being revoked and client sessions being logged out — users may need to log in again
      • The above can be avoided by moving the device_devices table into the account database manually
    • Guest registration can now be separately disabled with the new client_api.guests_disabled configuration option
    • Outbound connections now obey proxy settings from the environment, deprecating the federation_api.proxy_outbound configuration options

    Fixes

    • The roomserver input API will now strictly consume only one database transaction per room, which should prevent situations where the roomserver can deadlock waiting for database connections to become available
    • Room joins will now fall back to federation if the local room state is insufficient to create a membership event
    • Create events are now correctly filtered from federation /send transactions
    • Excessive logging when federation is disabled should now be fixed
    • Dendrite will no longer panic if trying to retire an invite event that has not been seen yet
    • The device list updater will now wait for longer after a connection issue, rather than flooding the logs with errors
    • The device list updater will no longer produce unnecessary output events for federated key updates with no changes, which should help to reduce CPU usage
    • Local device name changes will now generate key change events correctly
    • The sync API will now try to share device list update notifications even if all state key NIDs cannot be fetched
    • An off-by-one error in the sync stream token handling which could result in a crash has been fixed
    • State events will no longer be re-sent unnecessary by the roomserver to other components if they have already been sent, which should help to reduce the NATS message sizes on the roomserver output topic in some cases
    • The roomserver input API now uses the process context and should handle graceful shutdowns better
    • Guest registration is now correctly disabled when the client_api.registration_disabled configuration option is set
    • One-time encryption keys are now cleaned up correctly when a device is logged out or removed
    • Invalid state snapshots in the state storage refactoring migration are now reset rather than causing a panic at startup
    Source code(tar.gz)
    Source code(zip)
  • v0.6.3(Feb 10, 2022)

    Features

    • Initial support for m.login.token
    • A number of regressions from earlier v0.6.x versions should now be corrected

    Fixes

    • Missing state is now correctly retrieved in cases where a gap in the timeline was closed but some of those events were missing state snapshots, which should help to unstick slow or broken rooms
    • Fixed a transaction issue where inserting events into the database could deadlock, which should stop rooms from getting stuck
    • Fixed a problem where rejected events could result in rolled back database transactions
    • Avoided a potential race condition on fetching latest events by using the room updater instead
    • Processing events from /get_missing_events will no longer result in potential recursion
    • Federation events are now correctly generated for updated self-signing keys and signed devices
    • Rejected events can now be un-rejected if they are reprocessed and all of the correct conditions are met
    • Fetching missing auth events will no longer error as long as all needed events for auth were satisfied
    • Users can now correctly forget rooms if they were not a member of the room
    Source code(tar.gz)
    Source code(zip)
  • v0.6.2(Feb 4, 2022)

  • v0.6.1(Feb 4, 2022)

    Features

    • Roomserver inputs now take place with full transactional isolation in PostgreSQL deployments
    • Pull consumers are now used instead of push consumers when retrieving messages from NATS to better guarantee ordering and to reduce redelivery of duplicate messages
    • Further logging tweaks, particularly when joining rooms
    • Improved calculation of servers in the room, when checking for missing auth/prev events or state
    • Dendrite will now skip dead servers more quickly when federating by reducing the TCP dial timeout
    • The key change consumers have now been converted to use native NATS code rather than a wrapper
    • Go 1.16 is now the minimum supported version for Dendrite

    Fixes

    • Local clients should now be notified correctly of invites
    • The roomserver input API now has more time to process events, particularly when fetching missing events or state, which should fix a number of errors from expired contexts
    • Fixed a panic that could happen due to a closed channel in the roomserver input API
    • Logging in with uppercase usernames from old installations is now supported again (contributed by hoernschen)
    • Federated room joins now have more time to complete and should not fail due to expired contexts
    • Events that were sent to the roomserver along with a complete state snapshot are now persisted with the correct state, even if they were rejected or soft-failed
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Jan 28, 2022)

    Features

    • NATS JetStream is now used instead of Kafka and Naffka
      • For monolith deployments, a built-in NATS Server is embedded into Dendrite or a standalone NATS Server deployment can be optionally used instead
      • For polylith deployments, a standalone NATS Server deployment is required
      • Requires the version 2 configuration file — please see the new dendrite-config.yaml sample config file
      • Kafka and Naffka are no longer supported as of this release
    • The roomserver is now responsible for fetching missing events and state instead of the federation API
      • Removes a number of race conditions between the federation API and roomserver, which reduces duplicate work and overall lowers CPU usage
    • The roomserver input API is now strictly ordered with support for asynchronous requests, smoothing out incoming federation significantly
    • Consolidated the federation API, federation sender and signing key server into a single component
      • If multiple databases are used, tables for the federation sender and signing key server should be merged into the federation API database (table names have not changed)
    • Device list synchronisation is now database-backed rather than using the now-removed Kafka logs

    Fixes

    • The code for fetching missing events and state now correctly identifies when gaps in history have been closed, so federation traffic will consume less CPU and memory than before
    • The stream position is now correctly advanced when typing notifications time out in the sync API
    • Event NIDs are now correctly returned when persisting events in the roomserver in SQLite mode
      • The built-in SQLite was updated to version 3.37.0 as a result
    • The /event_auth endpoint now strictly returns the auth chain for the requested event without loading the room state, which should reduce spikes in memory usage
    • Filters are now correctly sent when using federated public room directories (contributed by S7evinK)
    • Login usernames are now squashed to lower-case (contributed by BernardZhao)
    • The logs should no longer be flooded with Failed to get server ACLs for room warnings at startup
    • Backfilling will now attempt federation as a last resort when trying to retrieve missing events from the database fails
    Source code(tar.gz)
    Source code(zip)
  • v0.5.1(Nov 16, 2021)

    Features

    • Experimental (although incomplete) support for joining version 8 and 9 rooms
    • State resolution v2 optimisations (close to 20% speed improvement thanks to reduced allocations)
    • Optimisations made to the federation /send endpoint which avoids duplicate work, reduces CPU usage and smooths out incoming federation
    • The sync API now consumes less CPU when generating sync responses (optimised SelectStateInRange)
    • Support for serving the .well-known/matrix/server endpoint from within Dendrite itself (contributed by twentybit)
    • Support for thumbnailing WebP media (contributed by hacktivista)

    Fixes

    • The /publicRooms handler now handles POST requests in addition to GET correctly
    • Only valid canonical aliases will be returned in the /publicRooms response
    • The media API now correctly handles max_file_size_bytes being configured to 0 (contributed by database64128)
    • Unverifiable auth events in /send_join responses no longer result in a panic
    • Build issues on Windows are now resolved (contributed by S7evinK)
    • The default power levels in a room now set the invite level to 50, as per the spec
    • A panic has been fixed when malformed messages are received in the key change consumers
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Aug 24, 2021)

    Features

    • Support for serverside key backups has been added, allowing your E2EE keys to be backed up and to be restored after logging out or when logging in from a new device
    • Experimental support for cross-signing has been added, allowing verifying your own device keys and verifying other user's public keys
    • Dendrite can now send logs to a TCP syslog server by using the syslog logger type (contributed by sambhavsaggi)
    • Go 1.15 is now the minimum supported version for Dendrite

    Fixes

    • Device keys are now cleaned up from the keyserver when the user API removes a device session
    • The M_ROOM_IN_USE error code is now returned when a room alias is already taken (contributed by nivekuil)
    • A bug in the state storage migration has been fixed where room create events had incorrect state snapshots
    • A bug when deactivating accounts caused by only reading the deprecated username field has been fixed
    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(Jul 26, 2021)

    Features

    • Support for room version 7 has been added
    • Key notary support is now more complete, allowing Dendrite to be used as a notary server for looking up signing keys
    • State resolution v2 performance has been optimised further by caching the create event, power levels and join rules in memory instead of parsing them repeatedly
    • The media API now handles cases where the maximum file size is configured to be less than 0 for unlimited size
    • The initial_state in a /createRoom request is now respected when creating a room
    • Code paths for checking if servers are joined to rooms have been optimised significantly

    Fixes

    • A bug resulting in cannot xref null state block with snapshot during the new state storage migration has been fixed
    • Invites are now retired correctly when rejecting an invite from a remote server which is no longer reachable
    • The DNS cache cache_lifetime option is now handled correctly (contributed by S7evinK)
    • Invalid events in a room join response are now dropped correctly, rather than failing the entire join
    • The prev_state of an event will no longer be populated incorrectly to the state of the current event
    • Receiving an invite to an unsupported room version will now correctly return the M_UNSUPPORTED_ROOM_VERSION error code instead of M_BAD_JSON (contributed by meenal06)
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Jul 12, 2021)

    Features

    • All-new state storage in the roomserver, which dramatically reduces disk space utilisation
      • State snapshots and blocks are now aggressively deduplicated and reused wherever possible, with state blocks being reduced by up to 15x and snapshot references being reduced up to 2x
      • Dendrite will upgrade to the new state storage automatically on the first run after upgrade, although this may take some time depending on the size of the state storage
    • Appservice support has been improved significantly, with many bridges now working correctly with Dendrite
      • Events are now correctly sent to appservices based on room memberships
      • Aliases and namespaces are now handled correctly, calling the appservice to query for aliases as needed
      • Appservice user registrations are no longer being subject to incorrect validation checks
    • Shared secret registration has now been implemented correctly
    • The roomserver input API implements a new queuing system to reduce backpressure across rooms
    • Checking if the local server is in a room has been optimised substantially, reducing CPU usage
    • State resolution v2 has been optimised further by improving the power level checks, reducing CPU usage
    • The federation API /send endpoint now deduplicates missing auth and prev events more aggressively to reduce memory usage
    • The federation API /send endpoint now uses workers to reduce backpressure across rooms
    • The bcrypt cost for password storage is now configurable with the user_api.bcrypt_cost option
    • The federation API will now use significantly less memory when calling /get_missing_events
    • MSC2946 Spaces endpoints have been updated to stable endpoint naming
    • The media API can now be configured without a maximum file size
    • A new dendrite-upgrade-test test has been added for verifying database schema upgrades across versions
    • Added Prometheus metrics for roomserver backpressure, excessive device list updates and federation API event processing summaries
    • Sentry support has been added for error reporting

    Fixes

    • Removed the legacy /v1 register endpoint. Dendrite only implements /r0 of the CS API, and the legacy /v1 endpoint had implementation errors which made it possible to bypass shared secret registration (thanks to Jakob Varmose Bentzen for reporting this)
    • Attempting to register an account that already exists now returns a sensible error code rather than a HTTP 500
    • Dendrite will no longer attempt to /make_join with itself if listed in the request server_names
    • /sync will no longer return immediately if there is nothing to sync, which happened particularly with new accounts, causing high CPU usage
    • Malicious media uploads can no longer exhaust all available memory (contributed by S7evinK)
    • Selecting one-time keys from the database has been optimised (contributed by S7evinK)
    • The return code when trying to fetch missing account data has been fixed (contributed by adamgreig)
    • Dendrite will no longer attempt to use /make_leave over federation when rejecting a local invite
    • A panic has been fixed in QueryMembershipsForRoom
    • A panic on duplicate membership events has been fixed in the federation sender
    • A panic has been fixed in in IsInterestedInRoomID (contributed by bodqhrohro)
    • A panic in the roomserver has been fixed when handling empty state sets
    • A panic in the federation API has been fixed when handling cached events
    Source code(tar.gz)
    Source code(zip)
  • v0.3.11(Mar 2, 2021)

    This is a security release to fix an identified vulnerability in SQLite deployments — if you are running Dendrite in SQLite mode, please upgrade immediately.

    Fixes

    • SECURITY: A bug in SQLite mode which could cause the registration flow to complete unexpectedly for existing accounts has been fixed (PostgreSQL deployments are not affected)
    • A panic in the federation sender has been fixed when shutting down destination queues
    • The /keys/upload endpoint now correctly returns the number of one-time keys in response to an empty upload request
    Source code(tar.gz)
    Source code(zip)
  • v0.3.10(Feb 17, 2021)

    Features

    • In-memory caches will now gradually evict old entries, reducing idle memory usage
    • Federation sender queues will now be fully unloaded when idle, reducing idle memory usage
    • The power_level_content_override option is now supported in /createRoom
    • The /send endpoint will now attempt more servers in the room when trying to fetch missing events or state

    Fixes

    • A panic in the membership updater has been fixed
    • Events in the sync API that weren't excluded from sync can no longer be incorrectly excluded from sync by backfill
    • Retrieving remote media now correcly respects the locally configured maximum file size, even when the Content-Length header is unavailable
    • The /send endpoint will no longer hit the database more than once to find servers in the room
    Source code(tar.gz)
    Source code(zip)
  • v0.3.9(Feb 4, 2021)

    Features

    • Performance of initial/complete syncs has been improved dramatically
    • State events that can't be authed are now dropped when joining a room rather than unexpectedly causing the room join to fail
    • State events that already appear in the timeline will no longer be requested from the sync API database more than once, which may reduce memory usage in some cases

    Fixes

    • A crash at startup due to a conflict in the sync API account data has been fixed
    • A crash at startup due to mismatched event IDs in the federation sender has been fixed
    • A redundant check which may cause the roomserver memberships table to get out of sync has been removed
    Source code(tar.gz)
    Source code(zip)
  • v0.3.8(Jan 28, 2021)

  • v0.3.7(Jan 26, 2021)

    Features

    • Sync filtering support (for event types, senders and limits)
    • In-process DNS caching support for deployments where a local DNS caching resolver is not available (disabled by default)
    • Experimental support for MSC2444 (Peeking over Federation) has been merged
    • Experimental federation support for MSC2946 (Spaces Summary) has been merged

    Fixes

    • Dendrite will no longer load a given event more than once for state resolution, which may help to reduce memory usage and database I/O slightly in some cases
    • Large well-known responses will no longer use significant amounts of memory
    Source code(tar.gz)
    Source code(zip)
Owner
matrix.org
A new basis for open, interoperable, decentralised real-time communication
matrix.org
Avatar generation library for GO language

GOvatar GOvatar is an avatar generation library written in GO Install To install the library and command-line program, use the following: $ go get -u

Oleg Lobanov 512 Aug 11, 2022
Go package captcha implements generation and verification of image and audio CAPTCHAs.

Package captcha ⚠️ Warning: this captcha can be broken by advanced OCR captcha breaking algorithms. import "github.com/dchest/captcha" Package captch

Dmitry Chestnykh 1.6k Aug 11, 2022
Procedural texture generation package.

Texture Generation A package for the procedural generation of textures. Based on the ideas contained in the Bryce 3D deep texture editor. More example

null 18 Jul 23, 2022
A Go-language library for the automatic generation of image collages.

CollageCreator is a Go-language library for the automatic generation of image collages.

August Schwerdfeger 0 Jan 29, 2022
Simple ANSi to PNG converter written in pure Go

AnsiGo Description AnsiGo is a simple ANSi to PNG converter written in pure Go. It converts files containing ANSi sequences (.ANS) into PNG images. Fo

Frederic Cambus 51 May 17, 2022
Super fast static photo and video gallery generator (written in Go and HTML/CSS/native JS)

fastgallery Fast static photo and video gallery generator Super fast (written in Go and C, concurrent, uses fastest image/video libraries, 4-8 times f

Toni Melisma 24 May 25, 2022
Storage and image processing server written in Go

Mort An S3-compatible image processing server written in Go. Still in active development. Features HTTP server Resize, Rotate, SmartCrop Convert (JPEG

Marcin Kaciuba 459 Aug 12, 2022
An image resizing server written in Go

picfit picfit is a reusable Go server to manipulate images (resize, thumbnail, etc.). It will act as a proxy on your storage engine and will be served

Florent Messa 1.7k Aug 9, 2022
A path tracer written in Go.

pt: a golang path tracer This is a CPU-only, unidirectional path tracing engine written in Go. It has lots of features and a simple API. Features Supp

Michael Fogleman 2k Aug 6, 2022
Pixelizer is an efficient image pixelizer written in go

Pixelizer is an image pixelizer written in go. It allows very simple and intuitive CLI pixelization. Installation To install Pixelizer, you

Rewan_ 104 Jul 2, 2022
Gorched is terminal based game written in Go inspired by "The Mother of all games" Scorched Earth

Gorched Gorched is terminal based game written in Go inspired by "The Mother of all games" Scorched Earth. How the game looks like depends on your ter

Ján Čabala 326 Aug 4, 2022
GameBoy Color emulator written in golang.

?? Worldwide 日本語のドキュメントはこちら GameBoyColor emulator written in golang. This emulator can play almost all ROMs work without problems and has many feature

Akatsuki 567 Aug 5, 2022
Quake 2 Level Renderer written in Go and OpenGL

go-quake2 Quake 2 Map Renderer written in Go and OpenGL. Features Loads any BSP file from Quake 2 Free roam around the environment Renders only a smal

Samuel Yuan 24 Jul 19, 2022
Favicon service written in Go

favicon-service (besticon) This is a favicon service: Supports favicon.ico and apple-touch-icon.png Simple URL API Fallback icon generation Docker ima

Matthias Lüdtke 602 Aug 9, 2022
A captcha library written in golang

gocaptcha 一个简单的Go语言实现的验证码 图片实例 简介 基于Golang实现的图片验证码生成库,可以实现随机字母个数,随机直线,随机噪点等。可以设置任意多字体,每个验证码随机选一种字体展示。 实例 使用: go get github.com/lifei6671/gocaptcha/

Minho 170 Jun 28, 2022
A simple API written in Go that creates badges in SVG format, based on the requested route.

A simple API written in Go that creates badges in SVG format, based on the requested route. Those graphics can be used to style README.md files, or to add tags to webpages.

Toby 3 Jul 2, 2021
ColorX is a library to determine the most prominent color in an image written in golang

ColorX is a library to determine the most prominent color in an image. ColorX doesn't use any sort of complex algorithms to calculate the prominent color, it simply loops over the image pixels and returns the color that occurs the most.

Hesham Abourgheba 1 Nov 11, 2021
Pbm - Package ppm implements a Portable Bit Map (PBM) image decoder and encoder written in Go

Package pbm import "github.com/slashformotion/pbm" Package pbm implements a Portable Bit Map (PBM) image decoder and encoder. The supported image col

slashformotion 0 Jan 5, 2022
A simulation to see what's the result among normal people、rich-second generation、hard-working people

A simulation to see what's the result of competion among normal people、rich-second generation and hard-working people. 假设: 一个社会集体中有部分富二代,部分努力的人,多数是普通人

Myrainhua 0 Feb 20, 2022
A multi-pass compiler written in Go comprised of scanner, recursive-descent parser, generation of AST, intermediate representation (ILOC), and code generation (Armv8).

GoLite Project - Go Huskies! This is a project conducted and led in the course MPCS 51300 Compilers at the University of Chicago. In a group of two, w

ocd_with_naming 0 Jan 10, 2022
DevTodo the Second

DevTodo2 DevTodo2 is a command-line task management utility. Tasks are hierarchically organised, have priorities, and track creation and completion ti

Alec Thomas 84 Jul 22, 2022
A simple Cron library for go that can execute closures or functions at varying intervals, from once a second to once a year on a specific date and time. Primarily for web applications and long running daemons.

Cron.go This is a simple library to handle scheduled tasks. Tasks can be run in a minimum delay of once a second--for which Cron isn't actually design

Robert K 211 Aug 8, 2022
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.

Fortio Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio is also used by, among others, Me

Fortio (Φορτίο) 2.6k Aug 9, 2022
A simple Cron library for go that can execute closures or functions at varying intervals, from once a second to once a year on a specific date and time. Primarily for web applications and long running daemons.

Cron.go This is a simple library to handle scheduled tasks. Tasks can be run in a minimum delay of once a second--for which Cron isn't actually design

Robert K 211 Aug 8, 2022
Golang port of simdjson: parsing gigabytes of JSON per second

This is a Golang port of simdjson, a high performance JSON parser developed by Daniel Lemire and Geoff Langdale. It makes extensive use of SIMD instructions to achieve parsing performance of gigabytes of JSON per second.

High Performance, Kubernetes Native Object Storage 1.4k Aug 17, 2022
Subdomain scanner, asynchronous dns packets, use pcap to scan 1600,000 subdomains in 1 second

ksubdomain是一款基于无状态的子域名爆破工具,类似无状态端口扫描,支持在Windows/Linux/Mac上进行快速的DNS爆破,在Mac和Windows上理论最大发包速度在30w/s,linux上为160w/s。 hacking8信息流的src资产收集 https://i.hacking8

boyhack 453 Aug 8, 2022
There are 1,000,000 uatoms in an atom, a fact which I frequently forget and/or second-guess.

There are 1,000,000 uatoms in an atom, a fact which I frequently forget and/or second-guess. This repo contains a lot of programming to save myself a little bit of remembering.

Tess Rinearson 3 Nov 14, 2021
ncurses matrix/log app written in go to visualize chess problems.

dorrella/matrix-curses Matrix using ncurses and gbin/goncurses. Visual matrix based puzzles Install need libncurses-dev. Probably hard to run on windo

null 0 Jan 12, 2022
Sparse matrix formats for linear algebra supporting scientific and machine learning applications

Sparse matrix formats Implementations of selected sparse matrix formats for linear algebra supporting scientific and machine learning applications. Co

James Bowman 133 Aug 16, 2022