Hey @vaerh! Thank you for submitting the RFC!

Could you please clarify the main motivation for using vault as the backend? Did I get it right that it's mostly because it allows to store the state with sensitive data securely?

If security is the main concern, could you please check the RFC about state encryption - #297, maybe it covers your problem. Thanks!

Another note, it's worth checking if the Vault license change could potentially affect OpenTofu users who would want to use hosted Vault as the backend.

Yes #297 partially covers my request except for one point, we will still have to have additional shared storage deployed locally or an external data provider.
Thanks!

Hey @vaerh, thanks for submitting this! Also, good point @kislerdm!

One note, to set timeline expectations, we're generally waiting until the stable release is out with any meaningful enhancements.

Now regarding the RFC itself. The main thing I'm interested in here is understanding community interest and how much people would use this. A feature like this will require a maintenance effort long-term, so we should always weigh that.

In this case, I'm wondering what fraction of people moving to OpenTofu actually want to keep using Vault, and get even more locked-in to it. Curious of others' thoughts (esp. people who plan to use OpenTofu alongside Vault long-term!).

Ok, let's wait for the community's opinion.

I'd believe it likely that organizations utilizing Vault would be hard pressed to move (currently no plans to drop Vault on my end), but if they were looking for an alternative there is still good reason to consider this feature now that OpenBao has been announced.

• Repo: OpenBao
• Article detailing announcement: https://www.theregister.com/2023/12/08/hashicorp_openbao_fork/

The bigger issue here is that Terraform State can be relatively large - and depending on the remote vault configuration - could cause denial of service issues across the cluster - especially when using Consul as the storage medium.

Consul can't chunk writes, so Consul will bounce the whole storage through Raft on every update - which in particularly large cases will cause Consul servers to start missing heartbeats and result in leadership instability.

It's alleviated a bit if you use Vault's integrated storage by chunking, but still not ideal on data with frequent writes like tf state.

Hey :)

Apologies, I accidentality started working on this - I got fed up and wondered what it would be like - I hadn't realised there was an issue already, so apologies if it seems if I'm trying to bypass the "decision"/"rfc" (I'm not on purpose!).

I took a look at the consul implementation (without thinking too much) and inherently included the functionality for chunking.

There's plenty still to do, as noted on the PR (#1680), but I published it for at least others to see :)

For what it's worth, my use case is quite different and "state encryption" does not help - as suggested in the PR, my main reason for this is not about the "security" of the storage of the secret but rather:

• I do not use a cloud provider
• I use minio for S3, but use a command access key for pipelines
• I use Gitlab JWT auth for pipelines to vault, which gives fantastic support for isolating deployments
• Using this backend, each pipeline will easily only have access to it's own state

If I do complete the remaining work I foresee, if the PR is not accepted, I'd completely understand, I'd likely just keep it on a personal fork to use, as I'd really like it

Matt

Hey @MatthewJohn no worries, PoC PRs are always welcome. However, in this specific case I would wait for a decision on #382 before putting in any more work. The main reason is that the current setup requires a large amount of long-term commitment from the core team. That's mainly because once people depend on it, removing a backend will cause frustration, so the core team will basically be responsible for the ongoing maintenance of the backend. Ideally, if #382 gets accepted, that would enable the community to write and distribute backend implementations independent of the OpenTofu core.

For the time being, I would recommend using state encryption to ensure each pipeline only has access to its own state. It's not perfect for your use case, but it will prevent accidental cross-contamination of pipelines.

Hi, I don't see a problem with an alternative implementation for our idea. The only thing that I didn't put into my implementation is the chunking, but I didn't test this implementation on large state files.
In any case, I will be glad if we see this functional in OT in some form.

In the discussion on #1680 we realized that problematic licensing on HashiCorp's repository will require us to target OpenBao instead, with potential Vault support as side effect.

One commentary I think I might add is that K/V with Raft might have a smaller file size limit than you'd expect. I think this can be solved by using multiple entries, but if the state file exceeds 1MB (the default upper limit in HashiCorp Vault and OpenBao), you might need to ask users to tweak the server configuration to raise max_storage_entry.

Hi @cipherboy , chunking can be used (as done in #1680) and is already done for the consul remote state.

@MatthewJohn Ah cool :-) How do you do the chunking transactionally? Would external transactions similar to etcd be useful here?

Edit: Ah, you use CAS and don't write any data to metadata, that should be fine. Just something to be aware of, but writing additional metadata isn't CAS-able.