No chain of trust for release 4.0.2

[dvzrv]

/kind bug
(sorry, forgot to remove the one line)

Description

The chain of trust for podman releases is again broken with 4.0.2 (also see #10972) and prevents downstreams from packaging the release.

The release 4.0.2 has been signed by @lsm5 (9E33DD8704CC03E2DEB84D9A1C1EDD7CC7C3A0DD) and not by @mheon (B7DBDCA456F7335E91F1C25CD3624C551D0515C4). No signature of the latter on the former key exists (FWIW, this is also still the case for the key of @ashley-cui as mentioned in the linked ticket above).

This topic has also been brought up in containers/netavark#231 and containers/aardvark-dns#83 and it will continue to be an issue unless the team working on this figures out a way to transparently cross-sign PGP keys and/or establish a common workflow to transparently assign developers and their keys the "right to release".
Most of this has already been mentioned in detail in the other tickets so I will abstain from being more specific. If you could look into this, it would be most appreciated. Thanks!

Steps to reproduce the issue:

1. git clone https://github.com/containers/podman

2. git verify-tag v4.0.2

Describe the results you received:

No chain of trust for signed tag.

Describe the results you expected:

Chain of trust for signed tag.

Additional information you deem important (e.g. issue happens only occasionally):

n/a

Output of podman version:

4.0.2

Output of podman info --debug:

n/a

Package info (e.g. output of rpm -q podman or apt list podman):

n/a

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

n/a

Additional environment details (AWS, VirtualBox, physical, etc.):

n/a

[lsm5]

@containers/podman-maintainers i think people cutting releases for all tools should spend some time signing each others keys.

[lsm5]

@mheon looks like keyserver hkps://keyserver.ubuntu.com still works. I haven't had any luck with the others.

[mheon]

I don’t have access to my key until Monday at earliest but I am fine with getting everyone qualified to release in a room to sign keys

…

On Mon, Mar 7, 2022 at 09:59 Lokesh Mandvekar ***@***.***> wrote: @mheon <https://github.com/mheon> looks like keyserver hkps:// keyserver.ubuntu.com still works. I haven't had any luck with the others. — Reply to this email directly, view it on GitHub <#13427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCGFQUPMELMEV7F6WWTU6YKU7ANCNFSM5P7UJENQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Foxboron]

You can sign each others keys or publish a signed document with one of the previous release keys. Both equally work for this usecase. Having some form of chain of trust is obviously preferably then keeping a document updated every now and then :)

Example from libvirt which had problems cross-signing keys during the pandemic https://libvirt.org/downloads.html#keys

[lsm5]

So, @mheon @baude @jwhonce and I (@lsm5) signed each others keys a few mins ago. Please give this a try again and let us know.

[lsm5]

@ashley-cui 's keys should be signed tomorrow

[dvzrv]

So, @mheon @baude @jwhonce and I (@lsm5) signed each others keys a few mins ago. Please give this a try again and let us know.

Thanks! Did you publish them (i.e. gpg --keyserver keyserver.ubuntu.com --send-keys 9E33DD8704CC03E2DEB84D9A1C1EDD7CC7C3A0DD)? The key does not yet have additional signatures when I get it from there.

[dvzrv]

On a related note: Please use signed tags (git tag -s <tag>) in the future again. The 4.0.2 tag is a lightweight tag (not signed, not annotated), where downstreams have to lock on to the commit (which is the signed object) the tag points at.

[baude]

Thanks! Did you publish them (i.e. gpg --keyserver keyserver.ubuntu.com --send-keys 9E33DD8704CC03E2DEB84D9A1C1EDD7CC7C3A0DD)? The key does not yet have additional signatures when I get it from there.

@lsm5 @mheon @jwhonce and I pushed to the keyserver yesterday, yes. As was mentioned earlier, @ashley-cui should get to hers today.

[Foxboron]

The keyserver.ubuntu.com isn't going to give you any signatures because of GDPR compliance, this is intended. I'm not sure what the best alternative is currently. It would be simpler if redhat put up the needed WKD infrastructure to host signing keys.

I'd suggest adding the keys to your webpage for download with signatures included.

Gnupg is terrible, sadly :/

[mheon]

Can I just say how much I hate GPG key distribution right now?

Plan B will be manually compiling all relevant signatures and then uploading ascii-armored pubkeys to somewhere trusted (probably each individual repo) to indicate folks trusted to make a release. Hopefully later this week.

[Foxboron]

Can I just say how much I hate GPG key distribution right now?

You are not alone! Arch recently bootstrapped the entire web of trust recently and made us independent on keyserver infrastructure. Also working on embedding GPG keys into our package files so we don't have to deal with the keyserver issues.

Plan B will be manually compiling all relevant signatures and then uploading ascii-armored pubkeys to somewhere trusted (probably each individual repo) to indicate folks trusted to make a release. Hopefully later this week.

Is the intention that different people can release different software or do you have a list of people that can do releases for all your software?

I'm personally of the opinion that just having a subpage on podman.io with a signed statement saying who can release software like linked above, with the keys uploaded, would maybe be simpler then maintaining it across several of your repositories. But Plan B works fine if that is what your prefer.

[mheon]

We talked it over more, and we'll probably have a single repo under containers/ where we provide all the keys, who is authorized to release what. We can link that from all containers/ project readmes and podman.io.

[Foxboron]

For inspiration you can look at how kernel.org or Arch Linux manages this.

https://gitlab.archlinux.org/archlinux/archlinux-keyring

https://git.kernel.org/pub/scm/docs/kernel/pgpkeys.git/

[lukehinds]

I wonder if we could leverage sigstore here to help you, do you have links on how you perform you current signing?

[Foxboron]

I assume you are all busy so I'm wondering if @mheon could clear-text sign a document stating the release managers and their keyids. It should be good enough for the time being while you figure out a better way to distribute the keys :)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon can we get this fixed for podman 4.1?

[mheon]

Yes.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon was this fixed?

[mheon]

No, can handle this Monday/Tuesday/whenever we cut 4.1.1

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Has this been fixed?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon ???

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I believe this is fixed, please reopen if I am mistaken.

[Foxboron]

As a reference for the future, this was fixed with the release-keys repository.

https://github.com/containers/release-keys