Switch from Pulp to createrepo-agent for RPMs #976
Conversation
ros_buildfarm/templates/release/rpm/sync_packages_to_main_job.xml.em
Outdated
Show resolved
Hide resolved
| 'ssh $UPLOAD_HOST -- mkdir -p /tmp/upload-${BUILD_TAG}/', | ||
| 'xargs -a binarypkg/rpm_upload_args.txt -I % scp % $UPLOAD_HOST:/tmp/upload-${BUILD_TAG}/', | ||
| 'xargs -a binarypkg/rpm_import_args.txt ssh $UPLOAD_HOST -- createrepo-agent /var/repos/%s/building/%s/ --arch %s --invalidate-family --invalidate-dependants' % (os_name, os_code_name, arch), | ||
| 'ssh $UPLOAD_HOST -- rm -fr /tmp/upload-${BUILD_TAG}/', |
There was a problem hiding this comment.
There are four separate SSH connections being opened here. The overhead of the excess connections accounts for the majority of time it takes to do most package uploads entirely. It would be nice to either enable OpenSSH features that allow us to re-use connections here or switch to an approach that can both upload and execute a script on the host in the same connection. I've investigated the latter and found ways to make it happen, but none that I found straightforward or well supported.
There was a problem hiding this comment.
It seems like configuring a persistent control socket with ControlMaster and ControlPersist could get us what we need, if we do open a persistent control socket we just need to take care that we close it reliably on each job's completion.
https://manpages.ubuntu.com/manpages/focal/en/man5/ssh_config.5.html
| 'type': 'string', | ||
| 'name': 'repo_list_file', | ||
| 'description': 'A specific text file or files containing RPM base URLs to import from. The default is the ros_bootstrap repository.', | ||
| 'default_value': '/home/jenkins-agent/ros_bootstrap_rpm_urls.txt', |
There was a problem hiding this comment.
Generally in favor of not adding more yaml where it isn't required but does the url format for importing repositories allow us to specify the expected signing key for the repository?
Even if the answer is no, I don't consider this a review blocker since we don't have an active RPM bootstrap repository or an immediate need for one.
There was a problem hiding this comment.
Right now, createrepo-agent isn't even checking the upstream repository signatures. I have it on the TODO list in that repo. For local signature checking, the public key must be in the local keystore, but there isn't any association between the public key and the specific repository.
Reason number 19385619 I'd like to stop signing metadata and start signing packages like upstream Fedora and RHEL do.
There was a problem hiding this comment.
I'll probably ask this question offline as well but if we use a package signing approach does the repository utility do anything to validate the integrity of signed packages within it or is it meant to behave as a simple intermediary and expect that the downstream users are validating package integrity?
See #972