Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Bootstrappable FCMP++ #9440

Draft
wants to merge 118 commits into
base: master
Choose a base branch
from
Draft

[WIP] Bootstrappable FCMP++ #9440

wants to merge 118 commits into from

Conversation

tobtoht
Copy link
Contributor

@tobtoht tobtoht commented Aug 17, 2024 •
edited
Loading

This is a WIP (early stages) draft PR for reproducible release builds with Rust FFI. Please check back later.

Based on #9436 (FCMP++ integration) and #8929 (Bootstrappable builds).

Tools

  • Binutils 2.41
  • LLVM 17.0.6
  • Rust 1.77.1

Sub-PRs

The Road to FCMP++

trtf

(Updated: September 5)

To-do (incomplete):

  • Bump Android NDK to latest LTS
  • Bump FreeBSD SDK to 12.3
  • Bump Darwin llvm to 17
  • Bump binutils to 2.40
  • Don't build rustdoc
  • Get builds working for all targets
  • Remove blobs from rustc source
  • Fix non-determinism on aarch64
  • Bundle fcmp_pp_rust deps
  • Split into sub-PRs
  • Fix depends CI
  • Test all builds
Target Builds Reproducible Bootstrappable Tested
x86_64-linux-gnu 🮱 🮱 🮱
arm-linux-gnueabihf 🮱 🮱 🮱
aarch64-linux-gnu 🮱 🮱 🮱
riscv64-linux-gnu 🮱 🮱 🮱
i686-linux-gnu 🮱 🮱 🮱
i686-w64-mingw32 🮱 🮱 🮱
x86_64-w64-mingw32 🮱 🮱 🮱
x86_64-unknown-freebsd 🮱 🮱 🮱 [1]
x86_64-apple-darwin 🮱 🮱 🮱 [2]
aarch64-apple-darwin 🮱 🮱 🮱 [2]
arm-linux-androideabi 🮱 🮱 ❌ [3]
aarch64-linux-android 🮱 🮱 ❌ [3]

[1] = links against precompiled freebsd system libraries
[2] = links against precompiled libraries from the darwin SDK
[3] = built with a precompiled toolchain from the Android NDK (there is currently no way around this)

j-berman and others added 30 commits July 29, 2024 03:40
@j-berman
rough fcmp++ tree impl (lots of work remaining to clean it up and fix)
e1c03f4
@j-berman
remove whitespaces
1ba876b
@j-berman
test validates lowest layer in tree
9e6b93b
@j-berman
fix c1 c2 layer indexing issue in test helper get_last_chunk
33ad50b
@j-berman
actual indexing fix, tests now passing
c05bd80
@j-berman
Cleaner file organization
ad8872a
@j-berman
template all curve_trees types & funcs, rename tower_cycle_types to t…
5103a94 
…ower_cycle
@j-berman
Implement CurveTrees & CurveTreesUnitTest classes to simplify callers
d9390c7
@j-berman
Add Curve class, and Helios & Selene classes that derive from Curve
29e0fe7
@j-berman
Use widths from fcmp++ repo test & align tests with width
9e68475
@j-berman
slight simplification to CurveTrees::hash_layer
26009ba
@j-berman
Consolidate hash_leaf_layer into hash_layer
4ade675
@j-berman
Move curve_trees.h implementations into curve_trees.cpp file
9ba00be
@j-berman
Cleaner template usage, moved static functions out of CurveTrees class
5ad0269
@j-berman
small cleanup
e68ea2e
@j-berman
start LMDB grow_tree
af9b74f
@kayabaNerve @j-berman
Remove cxx and expose scalars/points directly
af47a13
@kayabaNerve @j-berman
fmt, clippy
42f6ef2
@kayabaNerve @j-berman
Use statics on the Rust side for generators
c792b21
@kayabaNerve @j-berman
Only pass a single prior child
effa9ee
@j-berman
don't want to expose generator lengths in ffi
517f5a3
@j-berman
explicit type response to hash_grow 's
a1ee603
@kayabaNerve @j-berman
Remove reference from m_hash_init_point
988c4ea 
Identified by kayabaNerve, patch suggested by j-berman.
@j-berman
Simplify edge case handling in hash_layer
ab7c741 
- When retrieving last chunks, set next_start_child_chunk_index
so can know the correct start index without needing to modify
the offset
- Other smaller cleanup
@j-berman
Implement get_tree_last_chunks in db, can now extend tree in db
1ffde09
@j-berman
implement db->audit_tree, and flesh out db test to init/extend tree
6045357
@j-berman
cleaner lmdb test structure for curve trees
17b1f42
@j-berman
CurveTreesUnitTest -> CurveTreesGlobalTree class
c7c6c6a
@j-berman
Set up trim_tree_in_memory test
ae89fdd
@j-berman
expose and test hash_trim from rust lib
e8af709
kayabaNerve
kayabaNerve reviewed Aug 20, 2024
View reviewed changes
contrib/depends/packages/rustc.mk Outdated
rm -rf src/llvm-project && \
find . -type f -regex ".*\.\(a\|dll\|exe\|lib\)$$$$" -delete && \
find . -type f -name ".cargo-checksum.json" -print0 | xargs -0 -I% sh -c 'echo "{\"files\":{}}" > "%"' && \
find . -type f -name "Cargo.lock" -delete && \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this line delete the lockfile which enables deterministic builds by specifying the exact versions of all Rust dependencies to use?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the Cargo.lock file is necessary because we're removing binaries from the source code, causing checksum mismatches.

Builds are still reproducible. All dependencies are vendored, and we don't have network access here. Version numbers are included in Cargo.toml.

This PR isn't ready for review.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I didn't mean to criticize a WIP. I just wanted clarity, or to point out an error which would presumably make your life more difficult. I now have the former.

My one other chime in is the version numbers in the toml aren't exact, yet ranges, but sounds like that won't be an issue.

@MrCyjaneK
Copy link

MrCyjaneK commented Aug 21, 2024 •
edited
Loading

Hey - I've seen that NDK task is unchecked, you can grab latest NDK patch from here: https://github.com/MrCyjaneK/monero_c/blob/master/patches/monero/0013-android-ndk-version-bump.patch and https://github.com/MrCyjaneK/monero_c/blob/master/patches/monero/0013-rename-arm-armv7a.patch it cointains some hacks - but hopefully will be useful.

This was referenced Aug 25, 2024
Open
Draft
Draft
@tobtoht tobtoht mentioned this pull request Sep 2, 2024
Draft
@j-berman
Fix compile on arm-linux-androideabi (32-bit) using a newer NDK
41b1985 
- @tobtoht reported err with size_t -> uint64_t
- Also address some PR comments (@vtnerd namespace comment +
@Boog900 freeing unallocated rust type)
- Some cleaning
@tobtoht tobtoht mentioned this pull request Sep 3, 2024
Draft
14 tasks
tobtoht and others added 16 commits September 5, 2024 13:55
@tobtoht
Bootstrappable Builds
57068ca
@tobtoht
guix: add rust toolchain, cargo fetch rust deps
0a286de
@tobtoht
guix: aarch64-linux-gnu Rust FFI cross-compilation [WIP]
fa1dc62
@tobtoht
guix: build more targets
923959c
@tobtoht
guix: windows builds
bedf302
@tobtoht
guix: use depends to build rustc, bundle fcmp_pp_rust deps
d0e338e
@tobtoht
depends: rustc: remove blobs
66f3c55
@tobtoht
guix: fix i686-pc-windows-gnu build
3a98644 
Might be easier to just build rust std with panic=abort?
@tobtoht
guix: x86_64-unknown-freebsd builds
45901fc
@tobtoht @MrCyjaneK
guix: android builds
80a444e 
Co-authored-by: Czarek Nakamoto <[email protected]>
@tobtoht
guix: fix arm-linux-androideabi build
619d853
@tobtoht
depends: android_ndk: bump to 27 (latest LTS)
02aff2e
@tobtoht
guix: darwin builds
7496249
@tobtoht
guix: bump time-machine to 0ba20c76349709fdb04baa8d263e3c0c7bd7b4e7
a3eb1dd
@tobtoht
depends: libusb: temp disable patch
4c52eca
@tobtoht
fcmp_pp: remove android fix
d238c24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kayabaNerve kayabaNerve kayabaNerve left review comments

Assignees
No one assigned
Labels
build system critical important in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@tobtoht @kayabaNerve @MrCyjaneK @0xFFFC0000 @j-berman