Dependencies versioning issues

[dryajov]

Dependencies versioning issues

Nimble issues summary

• Nimble will look at the list of tags and install the latest tagged version for the range
• If, the repository isn't tagged, nimble will simply download head and use that version
• Not having tags, breaks semantic versioning
• Most status packages aren't tagged
• It's possible to pin version to specific commits, but this still breaks semantic versioning on other leaf dependencies
• In short, nimble and the fact that packages aren't tagged properly, breaks versioning and dependencies across the board
• Lock files are introduced to deal with this, lock files "pin" the dependency to a specific commit and version
  • Still, this doesn't address the issue with existing non-tagged dependencies

Nimble pre lock files (< 0.14.2)

Nimble install policy is as follows:

---

Versions

Versions of cloned packages via Git or Mercurial are determined through the
repository's tags.

When installing a package which needs to be downloaded, after the download is
complete and if the package is distributed through a VCS, Nimble will check the
cloned repository's tags list. If no tags exist, Nimble will simply install the
HEAD (or tip in Mercurial) of the repository. If tags exist, Nimble will attempt
to look for tags which resemble versions (e.g. v0.1) and will then find the
latest version out of the available tags, once it does so it will install the
package after checking out the latest version.

You can force the installation of the HEAD of the repository by specifying
#head after the package name in your dependency list.

---

However, due to several issues with nimble itself, which invalidated versioning, tagging has been neglected across many packages, this is the case with most status nim packages. This breaks any sort of semantic versioning.

Nimble with lock files (>= 0.14.2)

---

nimble lock

The nimble lock command will generate or update a package lock file named
nimble.lock. This file is used for pinning the exact versions of the
dependencies of the package. The file is intended to be committed and used by
other developers to ensure that exactly the same version of the dependencies is
used by all developers.

Currently the lock file have the structure as in the following example:

{
  "version": 1,
  "packages": {
     ...
     "chronos": {
      "version": "3.0.2",
      "vcsRevision": "aab1e30a726bb47c5d3f4a75a826981836cde9e2",
      "url": "https://github.com/status-im/nim-chronos",
      "downloadMethod": "git",
      "dependencies": [
        "stew",
        "bearssl",
        "httputils",
        "unittest2"
      ],
      "checksums": {
        "sha1": "a1cdaa77995f2d1381e8f9dc129594f2fa2ee07f"
      }
    },
    ...
  }
}

• version - JSON schema version.
• packages - JSON object containing JSON objects for all dependencies,
• chronos - Nested JSON object keys are the names of the dependencies
  packages.
• version - The version of the dependency.
• vcsRevision - The revision at which the dependency is locked.
• url - The URL of the repository of the package.
• downloadMethod - git or hg according to the type of the repository at
  url.
• dependencies - The direct dependencies of the package. Used for writing the
  reverse dependencies of the package in the nimbledata.json file. Those
  packages' names also must be in the lock file.
• checksums - A JSON compound object containing different checksums used for
  verifying that a downloaded package is exactly the same as the pinned in the
  lock file package. Currently, only sha1 checksums are supported
• sha1 - The sha1 checksum of the package files.

If a lock file nimble.lock exists, then on performing all Nimble commands
which require searching for dependencies and downloading them in the case they
are missing (like build, install, develop), it is read and its content is
used to download the same version of the project dependencies by using the URL,
download method and VCS revision written in it. The checksum of the downloaded
package is compared against the one written in the lock file. In the case the
two checksums are not equal then it will be printed error message and the
operation will be aborted. Reverse dependencies are added for installed locked
dependencies just like for any other package being locally installed.

---

Potential solutions

• Nimble with lock files
  • Seems sensible, given that this are officially supported
  • Still unclear how this work with semantic versioning with leaf dependencies
  • There are rumors that it has it's own set of issues - anyone has details on this?
• Nimbus build system
  • Status official build and dependency management system
  • Mostly used by top level dependencies/projects (nimbus, codex, waku)
    • When used with leaf dependencies, requires the use of EXCLUDED_NIM_PACKAGES on top level projects, to prevent nested vendor directory clashes
• libp2p pinning
  • Uses an ad-hock lock file mechanism - https://github.com/status-im/nim-libp2p/blob/unstable/libp2p.nimble#L118-L157
  • It's super simple
  • Unclear how it handles semantic versions in third party packages
    • Potentially same issue as the official lock file?

[benbierens]

If I understand correctly, a problem with the nimble lock-files solution is that (almost?) none of Status' repositories are properly tagged. In this case, I propose we could lead by example and properly tag the repositories under our control. Correct me if I'm wrong, but tagging them won't interfere with the Nimbus build system method we're using right now.

If nimble lock-files is the supported solution and will be in the future, then I propose we use it. However, I wouldn't suggest we decide to migrate right now for two reasons: I'd like to know all about those previously mentioned set of issues. And I'd like someone much more experienced with nim/nimble and the nimbus build system to do perhaps a time-boxed trial run.

[dryajov]

I agree on both points, we should start tagging our stuff more diligently and attempt to slowly move some of the projects to nimble lock files, in some shape or form.

[zah]

The lack of tagged revisions is not really a problem for migrating to Nimble lock files IMHO (although adding tags is still a worthy goal).

The key functionality to understand here is the new "develop mode" of Nimble which allows to work with your in-house dependencies in pretty much the same way as you currently do with the vendor folder. I've explained how this works here:

https://forum.nim-lang.org/t/6738#41857

We have some prepared scripts which can help a team migrate from NBS to Nimble (allowing both systems to be used during a transition period).

[elcritch]

After today's meeting we seemed to reach a consensus that we can do both Nimble lock files and the Nimbus build system. It's a bit more work, but along with begging to tag releases will get us better positioned to use semantic versioning with Nimble files in the future.

Initially it seems we'd want use the Nimbus build system for CI, but ensure that the Nimble file and Nimble lock files are consistent with the vendored versions. I have a small utility that can sync them pretty easily.

[elcritch]

Updated Thoughts!

Note that yesterday I ended up changing my thinking a bit. Originally I was thinking mainly about Nimble lockfiles. Looking more at the Nimbus build system with vendor/ folders, I think that Atlas aligns better with Nimbus and our work flows long term.

We can still use Nimble lock files, however, likely with a utility to generate Nimble lock files from the Nimbus build system. Doing this more for tagging deps and getting habits ready for SemVer.

Generally I think Atlas matches the Nimbus build system setup more naturally, but it's not yet ready as a full replacement. Skip to the last section for thoughts on using Atlas and Nimbus together. It's likely a matter of months before Atlas could be feature compatible with Nimbus build system and potentially fully replace Nimbus.

Background

Nimble

Nimble 0.14.* have been sorta of a pain, but lock files do work. However they don't work all to well with the old cd someDep; nimble -g workflow which gave a a way to setup workspaces of sorts.

The new nimble 0.14 workflow is more complicated due to how it verifies hashes on dependencies. It's a great idea! Unfortunately the implementation details and ecosystem present challenges when working with multiple deps.

I still don't understand how to use Nimble 0.14 with a workspace style setup ( ^_^" ).

Nimble 0.14 works on a Nimble directory structure looks like (Nimble 0.13 used pkgs):

~/.nimble/pkgs2/depA-0.3.4-jjfdfadfjkljlkfj/
~/.nimble/pkgs2/projectFoo-1.6.0-fdfjklfaffaafd/
~/.nimble/pkgs2/projectBar-0.4.0-fffdfjaskdljdl/

It's important to note that the folders in the Nimble pkgs2 like pkgs2/depA-0.3.4-jjfdfadfjkljlkfj/ are not the direct contents of the git repo. Only the files specified in the Nimble file get copied into the directory. This further complicates using Nimble in "workspace" mode.

Now in Nimble 0.13 you could setup a global develop link so you could create a sort of workspace:

~/.nimble/pkgs2/projectFoo-1.6.0/link -> ~/projs/codex_ws/projectFoo/src

Nimble 0.14 also verifies each dependency every time you build or run tests, which leads to difficulties when working on "dirty deps". The ability to work on these may have been added, but it's been somewhat fragile. I've backed out of using Nimble 0.14 in my Nim projects at least twice.

Pros of Nimble 0.14 are lockfiles and the ability to replicate builds much better. Especially with the lack of proper tagging on many Nim deps. However, the lockfiles must include both the Git commit hash and another hash of the project to work.

For developing single packages Nimble 0.14 works well! But it fails when working with workspaces of related dependencies.

Nimbus

With Nimbus build system we get a setup much more like a workspace:

projectFoo/
projectFoo/vendor/depA/
projectFoo/vendor/depB/
...
projectBar/
projectBar/vendor/depA/
projectBar/vendor/depB/
projectBar/vendor/projectFoo/
projectBar/vendor/...

Where projectFoo is a git repo with vendor/depA being normal git submodules. I refer to this as an "inverted workspace".

This works well for CI and reasonably well for working with multiple projects. You can go into the vendor/ module and use git to explore versions and manually update them, etc. It's a bit of a pain to update deps, but doesn't take that long if deps largely live on main/master branches.

Unfortunately, to build dependencies and such the Nimbus build system actually shims the nimble to emulate this and to setup the paths correctly.

Nimbus also uses a nimbus-build-system.paths file to specify for Nim which paths to use. This is very nice once you get used to it as it's very clear which paths are being used and set while not cluttering up the compiler invocation with the path args.

Atlas

Atlas seems to have been designed starting with some of the best points of the Nimbus build system. It's designed around workspaces first and all dependencies are cloned repos.

Atlas main style workspace looks like:

codex_ws/atlas.workspce
codex_ws/depA
codex_ws/depB
codex_ws/projectFoo/
codex_ws/projectBar/

Note that the atlas.workspace doesn't contain dependency information, but just configuration options for Atlas including the directories to use.

You use Atlas by going into the project directory codex_ws/projectFoo and running atlas install. This will generate a codex_ws/projectFoo/nim.cfg. Similar to the Nimbus build system, Nim will use that file to find all of the needed dependencies.

That means you can use regular nim c -r tests/testAll.nim to run compile projects, etc. Note you can add build tasks to codex_ws/projectFoo/config.nims and then use nim test which is recommended.

So far pretty simple. It can get a bit confusing as you need to remember that in this setup all projects share the same dependencies, which has pros and cons.

However, note that version resolution is pretty broken in Atlas still! It's being worked on, but essentially it currently will work like Nimble 0.13 and use the master/main branches.

Shared dependency versions in Atlas

Now if you go into another project codex_ws/projectBar/ and run atlas install it will clone any missing deps for projectBar and update the existing deps to match the version(s) used that projects projectBar.nimble.

A pro of this setup is that now you can go back to projectFoo and try running it with the dependency versions from projectBar. The con is if you forget this!

However, for a project like Codex with one master project, it's nice since you don't really need to worry about ensuring das-dht-emulator is using the same dependencies.

You can also switch quickly between versions since they're all Git repos by running atlas install again which will change any relevant commits.

Open problems remain for how Atlas can handle generated binary libraries when switching between dependency versions. Likely a "brute force" could suffice which would rebuild any libraries when the version is changed. This would be somewhat similar to changing deps with Nimbus build system.

Atlas Lock Files

Atlas supports lock files as well. So you can also do:

cd codex_ws/projectFoo/
atlas lock atlas.lock
atlas replay atlas.lock

To set this project back to it's deps. Running atlas replay will essentially run git checkout COMMIT_ID for each module dependency.

Integrating Nimbus Build System with Atlas

I noted that Nimbus build system and Atlas were more compatible in the first paragraph, but so far they seem fairly different. However, I recently added a PR to Atlas to add support for "inverted workspaces". In this mode you can setup Atlas workspace to look like:

projectBar/
projectBar/vendor/atlas.workspace
projectBar/vendor/depA/
projectBar/vendor/depB/
projectBar/vendor/projectFoo/
projectBar/vendor/...

Basically Atlas can work with a Nimbus vendor folder! In this case workspace="vendor" in the Atlas workspace file.

With this setup you can run atlas replay atlas.lock and it will setup the vendor submodules to match that of the lock file.

Another useful command is atlas pin depA which will examine the depA repo and update the atlas lockfile to match that repos current commit id. Eventually this could replace git submodules for vendoring.

These features be useable with Nimbus vendor folder after my PR gets in. I've already been using Atlas in combination with Nimbus build system already. Once Atlas dependency resolution is working it'll let us have the benefits of using SemVer with the Nimbus build system.

Based on this it seems to me that Atlas will provide a much more natural upgrade from the Nimbus build system. It's possible Nimbus build could just integrate Atlas so as to retain the same workflows but simplify it's setup and usage.

[elcritch]

For those curious, here's the PR to add "inverted workspaces" in Atlas nim-lang/atlas#48

[dryajov]

Thanks for writing this down, definitely a good overview of how to eventually replace NBS with atlas. We do want to be consistent with the rest of the ecosystem.

[cskiraly]

Fantastic writeup, Thanks!

In my view part of the problem comes from naturally conflicting requirements:

• on one hand we love reproducible builds,
• on the other hand we like dependency resolution.

Reproducible builds:
A good way is to have git hashes of the whole dependency tree, e.g. using git submodules (like in NBS), or with repo, or with any other tool (lock file with has information).

• git hashes seem to be good for this (see https://git-scm.com/docs/hash-function-transition/)
• Some (e.g. Nimble 0.14+, but also Yocto as far as I remember) add a hash of the working dir to make this even stronger, or directly use the working dir hash instead of the git hash. This makes sense from the theoretical point of view of avoiding being bound to the SCM, but in practice it seems to me it creates more issues than problems it solves.
• clearly anything malleable, such as git tags or branch names are a no-go here. Please do not depend on "master" or "head"

Dependency resolution is instead about specifying a version range for dependant libraries.

• here version can mean many things, and range can mean many things
  • most know way is semantic versioning, which allows range definitions
  • one could also define git hash based range definitions, although I haven't seen it in practice
• Nimbus and the nim library ecosystem clearly fails on this, with most libraries not having version tags at all.

In my understanding, dependency resolution tries to find a single version for each library that matches all the range requirements, but of course these ranges (and dependencies) themselves depend on the choices made, so it is quite a problem. I'm not sure how well defined and deterministic algorithms are in this regard. From what I see in various systems that I use (apt, pip, nimble), this is not well understood.

To these two, we can add the requirement for:

• breaking free from the fixed versions for development (this is where I have NBS's reset of the code in an update)
• switching between branches together with dependant library versions
• easily working through code in the whole dependency chain (this is where things that check out the working tree without the .git are bad)

NBS is conceptually not bad, but if you look at the Makefiles, there are some scary things in there. Also it is not meant to handle dependency ranges. In general, I think the community would need much more automated tools for handling the relation between the requirement for reproducible builds and dependency resolution.
I have yet to see Atlas.

[elcritch]

Some (e.g. Nimble 0.14+, but also Yocto as far as I remember) add a hash of the working dir to make this even stronger, or directly use the working dir hash instead of the git hash. This makes sense from the theoretical point of view of avoiding being bound to the SCM, but in practice it seems to me it creates more issues than problems it solves.

Good point, in theory the hashes are nice but in practice add a lot of friction. Yocto really is the gold standard as I understand in reproducible builds, but ugh it's a pain to use. :) Usually there's other much lower hanging fruit than a Git hash attack. Though having secure hash checking as a separate make-verified-build would be great -- for normal testing and debug builds it adds overhead for no benefit, IMHO.

In my understanding, dependency resolution tries to find a single version for each library that matches all the range requirements, but of course these ranges (and dependencies) themselves depend on the choices made, so it is quite a problem. I'm not sure how well defined and deterministic algorithms are in this regard. From what I see in various systems that I use (apt, pip, nimble), this is not well understood.

I believe they can be deterministic and the maths behind them are well understood (SAT solvers). However, they're based on the quality of the inputs, so e.g. git tags for versions would make it fragile.

[zah]

@elcritch, I can tell from your write-up that you are likely not familiar with the intended way to setup "workspaces" in the new Nimble. Admittedly, the documentation for this is not quite adequate, but I can explain the details in a quick call.

The following Nim forum post is a good introduction, as well as the documentation for the nimble develop and nimble sync commands in the Nimble repo's README. We have also prepared a special repository aiming to help all Status teams migrate from nimbus-build-system to Nimble with some handy commands for synchronizing the lock files with the vendor folder contents during the transition period:

https://github.com/status-im/nim-workspace#sync-vendor-revisions-to-workspace

So let's have a discussion that will make sure you are making a fully informed decision here.

[elcritch]

That’d be great. It’s late for me now, but let’s find a time. I’m definitely not aware of how the nimble 0.14 workspaces should work and haven’t found too many docs about it.

[dryajov]

I'd like to participate as well...