Skip to content

Commit

Permalink
Binary fuse filter (#4389)
Browse files Browse the repository at this point in the history
# Description

Addresses #4393.

This PR adds a Binary Fuse Filter library and switches BucketListDB to
use the new filter.

The Binary Fuse Filter library is based on
[this](https://github.com/FastFilter/xor_singleheader) library, but I
ended up essentially rewriting it in C++. The original library had a bug
where some filter bytes were non-deterministic. While this did not
affect correctness or the false positive rate, we need determinism
because State Archival will hash these filters into the ledger.
Additionally, the library did not support 32 bit filters, which are
required for State Archival.

Currently a draft until the (non protocol breaking) associated XDR
changes are merged: stellar/stellar-xdr#195

While Binary Fuse Filters are necessary for State Archival, they also
significantly improve BucketListDB. Compared to the older bloom filter
implementation, Bucket indexing time has decreased by 67% (which is
blocking on startup in some cases), decreases index memory size by 20%,
and reduces redundant disk reads by ~130x.

# Checklist
- [x] Reviewed the
[contributing](https://github.com/stellar/stellar-core/blob/master/CONTRIBUTING.md#submitting-changes)
document
- [x] Rebased on top of master (no merge commits)
- [x] Ran `clang-format` v8.0.0 (via `make format` or the Visual Studio
extension)
- [x] Compiles
- [x] Ran all tests
- [ ] If change impacts performance, include supporting evidence per the
[performance
document](https://github.com/stellar/stellar-core/blob/master/performance-eval/performance-eval.md)
  • Loading branch information
dmkozh committed Aug 9, 2024
2 parents f25405b + fb5d9b5 commit 55ec348
Show file tree
Hide file tree
Showing 18 changed files with 990 additions and 751 deletions.
1 change: 0 additions & 1 deletion Builds/VisualStudio/stellar-core.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -873,7 +873,6 @@ exit /b 0
<ClCompile Include="src\$(Configuration)\generated\xdr\XDRFilesSha256.cpp" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\lib\bloom_filter.hpp" />
<ClInclude Include="..\..\lib\catch.hpp" />
<ClInclude Include="..\..\lib\clara.hpp" />
<ClInclude Include="..\..\lib\util\finally.h" />
Expand Down
3 changes: 0 additions & 3 deletions Builds/VisualStudio/stellar-core.vcxproj.filters
Original file line number Diff line number Diff line change
Expand Up @@ -1754,9 +1754,6 @@
<ClInclude Include="..\..\src\util\xdrquery\XDRQueryParser.h">
<Filter>util\xdrquery</Filter>
</ClInclude>
<ClInclude Include="..\..\lib\bloom_filter.hpp">
<Filter>lib</Filter>
</ClInclude>
<ClInclude Include="..\..\lib\clara.hpp">
<Filter>lib</Filter>
</ClInclude>
Expand Down
14 changes: 7 additions & 7 deletions COPYING
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,14 @@ lib/asio:
Licensed under the Boost Software License 1.0
(http://www.boost.org/LICENSE_1_0.txt)

lib/bloom_filter.hpp:
lib/binaryfusefilter.h:

The C++ Open Bloom Filter Library,
Copyright 2000 Arash Partow
(arash at partow dot net)
(https://www.partow.net/programming/bloomfilter/index.html)
Licensed under the MIT license
(http://opensource.org/licenses/MIT)
The FastFilter library,
Copyright 2023 Daniel Lemire
(daniel at lemire dot me)
(<https://github.com/FastFilter/xor_singleheader>)
Licensed under the Apache License, Version 2.0
(<https://www.apache.org/licenses/LICENSE-2.0>)

lib/catch.hpp:

Expand Down
47 changes: 23 additions & 24 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ members = ["src/rust", "lib/tracy-client-sys"]

[patch.crates-io]
tracy-client-sys = { path = "lib/tracy-client-sys" }
stellar-xdr = { git = "https://github.com/stellar/rs-stellar-xdr", rev = "ae805d0f8c28ca86327a834eea9ce7d29b0a63bb" }

#[patch."https://github.com/stellar/rs-soroban-env"]
#soroban-env-common = { path = "../rs-soroban-env/soroban-env-common" }
Expand All @@ -13,7 +14,6 @@ tracy-client-sys = { path = "lib/tracy-client-sys" }
# stellar-xdr = { path = "../rs-stellar-xdr/" }

# [patch.crates-io]
# stellar-xdr = { git = "https://github.com/stellar/rs-stellar-xdr", rev = "2775f4b6" }
# wasmi = { package = "soroban-wasmi", git = "https://github.com/stellar/wasmi", rev = "862b32f5" }
# soroban-env-common = { git = "https://github.com/stellar/rs-soroban-env", rev = "eda2ab70" }
# soroban-native-sdk-macros = { git = "https://github.com/stellar/rs-soroban-env", rev = "eda2ab70" }
Expand Down
Loading

0 comments on commit 55ec348

Please sign in to comment.