Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(su): implement SU datastore to run purely on rocksdb #987

Open
VinceJuliano opened this issue Aug 29, 2024 · 0 comments
Open

perf(su): implement SU datastore to run purely on rocksdb #987

VinceJuliano opened this issue Aug 29, 2024 · 0 comments
Assignees
Labels
su ao Scheduler Unit

Comments

@VinceJuliano
Copy link
Collaborator

Background

Under heavy load, the SU database implementation on postgres runs into performance issues. When there are a lot of writes going on the writes become very slow etc... RocksDB was added originally because on the message list query because it contains json data which was stored in postgres, was very slow for large amounts of data.

Solution

To solve this, we can add another RocksDB directory into the SU that will hold the index information. Then we can remove postgres.

  1. implement a new Data Store that runs only on RocksDB. 1 Rocks database for the binary data which already exists for the messages and runs in Blob mode for large data, and another Rocks directory that holds the index information and runs in the regular mode.

  2. Write another binary like the migration binary to migrate the index information. This should be much faster then the one that migrates the Message bundles themselves because it only needs to grab index info which is quicker to fetch from postgres.

@VinceJuliano VinceJuliano added the su ao Scheduler Unit label Aug 29, 2024
@VinceJuliano VinceJuliano self-assigned this Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
su ao Scheduler Unit
Projects
None yet
Development

No branches or pull requests

1 participant