Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bootstrappable Builds #8929

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 0 additions & 49 deletions .github/workflows/gitian.yml

This file was deleted.

10 changes: 10 additions & 0 deletions .github/workflows/guix
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Workaround for: https://bugs.launchpad.net/ubuntu/+source/guix/+bug/2064115

abi <abi/4.0>,
include <tunables/global>

profile guix /usr/bin/guix flags=(unconfined) {
userns,

include if exists <local/guix>
}
92 changes: 92 additions & 0 deletions .github/workflows/guix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: ci/gh-actions/guix

on:
push:
paths:
- 'contrib/depends/**'
- 'contrib/guix/**'
- '!contrib/**.md'
- '.github/workflows/guix.yml'
pull_request:
paths:
- 'contrib/depends/**'
- 'contrib/guix/**'
- '!contrib/**.md'
- '.github/workflows/guix.yml'

jobs:
cache-sources:
runs-on: ubuntu-24.04
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we switch to debian (12)?

Copy link
Contributor Author

@tobtoht tobtoht Aug 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that might fix the AppArmor issue. Good suggestion, will try.

Nvm, GitHub only has Linux runners for Ubuntu.

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: depends sources cache
id: cache
uses: actions/cache@v4
with:
path: contrib/depends/sources
key: sources-${{ hashFiles('contrib/depends/packages/*') }}
- name: download depends sources
if: steps.cache.outputs.cache-hit != 'true'
run: make -C contrib/depends download

build-guix:
runs-on: ubuntu-24.04
needs: [cache-sources]
strategy:
fail-fast: false
matrix:
toolchain:
- target: "x86_64-linux-gnu"
- target: "aarch64-linux-gnu"
- target: "arm-linux-gnueabihf"
- target: "riscv64-linux-gnu"
- target: "i686-linux-gnu"
- target: "x86_64-w64-mingw32"
- target: "i686-w64-mingw32"
- target: "x86_64-unknown-freebsd"
- target: "x86_64-apple-darwin"
- target: "aarch64-apple-darwin"
- target: "aarch64-linux-android"
- target: "arm-linux-androideabi"

name: ${{ matrix.toolchain.target }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive
- name: remove bundled packages
run: sudo rm -rf /usr/local
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick question. Do we need this if we are building with GUIX? Since the premise of GUIX is sandboxed builds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just to free up disk space. See monero-project/monero-gui#4223

- name: depends sources cache
uses: actions/cache/restore@v4
with:
path: contrib/depends/sources
key: sources-${{ hashFiles('contrib/depends/packages/*') }}
- name: install dependencies
run: sudo apt update; sudo apt -y install guix git ca-certificates apparmor-utils
- name: fix apparmor
run: sudo cp .github/workflows/guix /etc/apparmor.d/guix; sudo /etc/init.d/apparmor reload; sudo aa-enforce guix || true
- name: purge apparmor
run: sudo apt purge apparmor
- name: build
run: SUBSTITUTE_URLS='http://bordeaux.guix.gnu.org' HOSTS="${{ matrix.toolchain.target }}" ./contrib/guix/guix-build
- uses: actions/upload-artifact@v4
with:
name: ${{ matrix.toolchain.target }}
path: |
guix/guix-build-*/output/${{ matrix.toolchain.target }}/*
guix/guix-build-*/logs/${{ matrix.toolchain.target }}/*
bundle-logs:
runs-on: ubuntu-24.04
needs: [build-guix]
steps:
- uses: actions/download-artifact@v4
with:
merge-multiple: true
- uses: actions/upload-artifact@v4
with:
name: "logs"
path: '**/logs/**'
6 changes: 2 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,8 @@ miniupnpcstrings.h
version/
ClangBuildAnalyzerSession.txt

# gitian
contrib/gitian/builder/
contrib/gitian/docker/
contrib/gitian/sigs/
# guix
/guix
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished builds are available in ./guix/guix-build-<COMMIT>/

├── guix-build-222ea1a30058
│   ├── build
│   │   └── distsrc-222ea1a30058-x86_64-linux-gnu  # build directory
│   ├── logs
│   │   └── x86_64-linux-gnu  # various logs for reproducibility debugging
│   │       ├── depends-gen_id-build.txt
│   │       ├── depends-gen_id-host.txt
│   │       ├── depends-hashes.txt
│   │       ├── depends-packages.txt
│   │       ├── guix-env.txt
│   │       ├── guix-hashes.txt
│   │       └── SHA256SUMS.part
│   ├── output
│   │   └── x86_64-linux-gnu
│   │       └── monero-x86_64-linux-gnu-222ea1a30058.tar.bz2  # binary tarball
│   └── var
│       ├── precious_dirs  # directories to keep when running `./contrib/guix/guix-clean`
│       └── profiles


# Created by https://www.gitignore.io

Expand Down
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1113,7 +1113,7 @@ if(MINGW)
elseif(APPLE OR OPENBSD OR ANDROID)
set(EXTRA_LIBRARIES "")
elseif(FREEBSD)
set(EXTRA_LIBRARIES execinfo)
set(EXTRA_LIBRARIES execinfo elf)
elseif(DRAGONFLY)
find_library(COMPAT compat)
set(EXTRA_LIBRARIES execinfo ${COMPAT})
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Portions Copyright (c) 2012-2013 The Cryptonote developers.
- [Release staging schedule and protocol](#release-staging-schedule-and-protocol)
- [Compiling Monero from source](#compiling-monero-from-source)
- [Dependencies](#dependencies)
- [Gitian builds](#gitian-builds)
- [Guix builds](#guix-builds)
- [Internationalization](#Internationalization)
- [Using Tor](#using-tor)
- [Pruning](#Pruning)
Expand Down Expand Up @@ -599,9 +599,9 @@ USE_DEVICE_TREZOR=OFF make release

For more information, please check out Trezor [src/device_trezor/README.md](src/device_trezor/README.md).

### Gitian builds
### Guix builds

See [contrib/gitian/README.md](contrib/gitian/README.md).
See [contrib/guix/README.md](contrib/guix/README.md).

## Installing Monero from a package

Expand Down
11 changes: 8 additions & 3 deletions contrib/depends/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ include builders/$(build_os).mk
include builders/default.mk
include packages/packages.mk

ifeq ($(GUIX_ENVIRONMENT),)
build_id_string:=$(BUILD_ID_SALT)
build_id_string+=$(shell $(build_CC) --version 2>/dev/null)
build_id_string+=$(shell $(build_AR) --version 2>/dev/null)
Expand All @@ -98,6 +99,10 @@ $(host_arch)_$(host_os)_id_string+=$(shell $(host_AR) --version 2>/dev/null)
$(host_arch)_$(host_os)_id_string+=$(shell $(host_CXX) --version 2>/dev/null)
$(host_arch)_$(host_os)_id_string+=$(shell $(host_RANLIB) --version 2>/dev/null)
$(host_arch)_$(host_os)_id_string+=$(shell $(host_STRIP) --version 2>/dev/null)
else
build_id_string:=$(realpath $(GUIX_ENVIRONMENT))
$(host_arch)_$(host_os)_id_string:=$(realpath $(GUIX_ENVIRONMENT))
Comment on lines +103 to +104
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any change to manifest.scm invalidates the cache.

endif

packages += $($(host_arch)_$(host_os)_packages) $($(host_os)_packages)
native_packages += $($(host_arch)_$(host_os)_native_packages) $($(host_os)_native_packages)
Expand All @@ -111,7 +116,7 @@ $(host_arch)_$(host_os)_native_toolchain?=$($(host_os)_native_toolchain)
include funcs.mk

toolchain_path=$($($(host_arch)_$(host_os)_native_toolchain)_prefixbin)
final_build_id_long+=$(shell $(build_SHA256SUM) toolchain.cmake.in)
final_build_id_long+=:[sha256sum]:$(shell $(build_SHA256SUM) toolchain.cmake.in)
final_build_id+=$(shell echo -n "$(final_build_id_long)" | $(build_SHA256SUM) | cut -c-$(HASH_LENGTH))
$(host_prefix)/.stamp_$(final_build_id): $(native_packages) $(packages)
$(AT)rm -rf $(@D)
Expand All @@ -124,8 +129,8 @@ $(host_prefix)/.stamp_$(final_build_id): $(native_packages) $(packages)
$(host_prefix)/share/toolchain.cmake : toolchain.cmake.in $(host_prefix)/.stamp_$(final_build_id)
$(AT)@mkdir -p $(@D)
$(AT)sed -e 's|@HOST@|$(host)|' \
-e 's|@CC@|$(toolchain_path)$(host_CC)|' \
-e 's|@CXX@|$(toolchain_path)$(host_CXX)|' \
-e 's|@CC@|$(host_CC)|' \
-e 's|@CXX@|$(host_CXX)|' \
-e 's|@AR@|$(toolchain_path)$(host_AR)|' \
-e 's|@RANLIB@|$(toolchain_path)$(host_RANLIB)|' \
-e 's|@NM@|$(toolchain_path)$(host_NM)|' \
Expand Down
38 changes: 23 additions & 15 deletions contrib/depends/funcs.mk
Original file line number Diff line number Diff line change
@@ -1,18 +1,25 @@
define int_vars
#Set defaults for vars which may be overridden per-package
$(1)_cc=$($($(1)_type)_CC)
$(1)_cxx=$($($(1)_type)_CXX)
$(1)_objc=$($($(1)_type)_OBJC)
$(1)_objcxx=$($($(1)_type)_OBJCXX)
$(1)_ar=$($($(1)_type)_AR)
$(1)_ranlib=$($($(1)_type)_RANLIB)
$(1)_libtool=$($($(1)_type)_LIBTOOL)
$(1)_nm=$($($(1)_type)_NM)
$(1)_cflags=$($($(1)_type)_CFLAGS) $($($(1)_type)_$(release_type)_CFLAGS)
$(1)_cxxflags=$($($(1)_type)_CXXFLAGS) $($($(1)_type)_$(release_type)_CXXFLAGS)
$(1)_arflags=$($($(1)_type)_ARFLAGS) $($($(1)_type)_$(release_type)_ARFLAGS)
$(1)_ldflags=$($($(1)_type)_LDFLAGS) $($($(1)_type)_$(release_type)_LDFLAGS) -L$($($(1)_type)_prefix)/lib
$(1)_cppflags=$($($(1)_type)_CPPFLAGS) $($($(1)_type)_$(release_type)_CPPFLAGS) -I$($($(1)_type)_prefix)/include
$(1)_cc=$$($$($(1)_type)_CC)
$(1)_cxx=$$($$($(1)_type)_CXX)
$(1)_objc=$$($$($(1)_type)_OBJC)
$(1)_objcxx=$$($$($(1)_type)_OBJCXX)
$(1)_ar=$$($$($(1)_type)_AR)
$(1)_ranlib=$$($$($(1)_type)_RANLIB)
$(1)_libtool=$$($$($(1)_type)_LIBTOOL)
$(1)_nm=$$($$($(1)_type)_NM)
$(1)_cflags=$$($$($(1)_type)_CFLAGS) \
$$($$($(1)_type)_$$(release_type)_CFLAGS)
$(1)_cxxflags=$$($$($(1)_type)_CXXFLAGS) \
$$($$($(1)_type)_$$(release_type)_CXXFLAGS)
$(1)_arflags=$$($$($(1)_type)_ARFLAGS) \
$$($$($(1)_type)_$(release_type)_ARFLAGS)
$(1)_ldflags=$$($$($(1)_type)_LDFLAGS) \
$$($$($(1)_type)_$$(release_type)_LDFLAGS) \
-L$$($($(1)_type)_prefix)/lib
$(1)_cppflags=$$($$($(1)_type)_CPPFLAGS) \
$$($$($(1)_type)_$$(release_type)_CPPFLAGS) \
-I$$($$($(1)_type)_prefix)/include
Comment on lines +3 to +22
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delay expansion of package variables.

$(1)_recipe_hash:=
endef

Expand All @@ -37,6 +44,7 @@ endef

define int_get_build_recipe_hash
$(eval $(1)_all_file_checksums:=$(shell $(build_SHA256SUM) $(meta_depends) packages/$(1).mk $(addprefix $(PATCHES_PATH)/$(1)/,$($(1)_patches)) | cut -d" " -f1))
final_build_id_long+=:[$(1)_all_file_checksums]$(foreach checksum,$($(1)_all_file_checksums),$(shell echo ":$(checksum)")):
$(eval $(1)_recipe_hash:=$(shell echo -n "$($(1)_all_file_checksums)" | $(build_SHA256SUM) | cut -d" " -f1))
endef

Expand All @@ -46,7 +54,7 @@ $(eval $(1)_all_dependencies:=$(call int_get_all_dependencies,$(1),$($($(1)_type
$(foreach dep,$($(1)_all_dependencies),$(eval $(1)_build_id_deps+=$(dep)-$($(dep)_version)-$($(dep)_recipe_hash)))
$(eval $(1)_build_id_long:=$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type) $($(1)_build_id_deps) $($($(1)_type)_id_string))
$(eval $(1)_build_id:=$(shell echo -n "$($(1)_build_id_long)" | $(build_SHA256SUM) | cut -c-$(HASH_LENGTH)))
final_build_id_long+=$($(package)_build_id_long)
final_build_id_long+=:[recipe]:$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type):[deps]$(foreach dep,$($(1)_build_id_deps),$(shell echo ":$(dep)")):[$($(1)_type)_id]:$($($(1)_type)_id_string):
Comment on lines -49 to +57
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increase verbosity of final_build_id_long to ease debugging reproducibility issues.


#compute package-specific paths
$(1)_build_subdir?=.
Expand Down Expand Up @@ -267,4 +275,4 @@ $(foreach package,$(all_packages),$(eval $(call int_config_attach_build_config,$
$(foreach package,$(all_packages),$(eval $(call int_add_cmds,$(package))))

#special exception: if a toolchain package exists, all non-native packages depend on it
$(foreach package,$(packages),$(eval $($(package)_unpacked): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) ))
$(foreach package,$(packages),$(eval $($(package)_extracted): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) ))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_unpacked target doesn't exist.

62 changes: 52 additions & 10 deletions contrib/depends/hosts/darwin.mk
Original file line number Diff line number Diff line change
@@ -1,23 +1,65 @@
OSX_MIN_VERSION=10.13
OSX_SDK_VERSION=11.0
XCODE_VERSION=12.2
XCODE_BUILD_ID=12B45b
LD64_VERSION=609
ifeq (aarch64, $(host_arch))
CC_target=arm64-apple-$(host_os)
else
CC_target=$(host)
endif
darwin_CC=clang -target $(CC_target) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(host_prefix)/native/SDK/ -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -mlinker-version=$(LD64_VERSION) -B$(host_prefix)/native/bin/$(host)-
darwin_CXX=clang++ -target $(CC_target) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(host_prefix)/native/SDK/ -iwithsysroot/usr/include/c++/v1 -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -mlinker-version=$(LD64_VERSION) -stdlib=libc++ -B$(host_prefix)/native/bin/$(host)-

OSX_SDK=$(host_prefix)/native/SDK

darwin_native_toolchain=darwin_sdk native_cctools

clang_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang")
clangxx_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang++")

# Flag explanations:
#
# -mlinker-version
#
# Ensures that modern linker features are enabled. See here for more
# details: https://github.com/bitcoin/bitcoin/pull/19407.
#
# -B$(build_prefix)/bin
#
# Explicitly point to our binaries (e.g. cctools) so that they are
# ensured to be found and preferred over other possibilities.
#
# -isysroot$(OSX_SDK) -nostdlibinc
#
# Disable default include paths built into the compiler as well as
# those normally included for libc and libc++. The only path that
# remains implicitly is the clang resource dir.
#
# -iwithsysroot / -iframeworkwithsysroot
#
# Adds the desired paths from the SDK
#

darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH \
$(clang_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
-B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
-isysroot$(OSX_SDK) \
-isysroot$(OSX_SDK) -nostdlibinc \
-iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks

darwin_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH \
$(clangxx_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
-B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
-isysroot$(OSX_SDK) -nostdlibinc \
-iwithsysroot/usr/include/c++/v1 \
-iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks

darwin_CFLAGS=-pipe
darwin_CXXFLAGS=$(darwin_CFLAGS)
darwin_ARFLAGS=cr

darwin_release_CFLAGS=-O1
darwin_release_CFLAGS=-O2
darwin_release_CXXFLAGS=$(darwin_release_CFLAGS)

darwin_debug_CFLAGS=-O1
darwin_debug_CXXFLAGS=$(darwin_debug_CFLAGS)

darwin_native_toolchain=native_cctools darwin_sdk

darwin_cmake_system=Darwin
13 changes: 11 additions & 2 deletions contrib/depends/hosts/freebsd.mk
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
freebsd_CC=clang-8
freebsd_CXX=clang++-8
clang_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang")
clangxx_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang++")

freebsd_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH $(clang_prog) --target=$(host) --sysroot=$(host_prefix)/native -iwithsysroot/usr/include
freebsd_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH $(clangxx_prog) --target=$(host) -stdlib=libc++ --sysroot=$(host_prefix)/native \
-iwithsysroot/usr/include/c++/v1 -iwithsysroot/usr/include

freebsd_AR=ar
freebsd_RANLIB=ranlib
freebsd_NM=nm
Expand Down
Loading
Loading