pkg: Use filtered_formula to represent dependencies
#10918
Conversation
There was a problem hiding this comment.
Some tests expectedly fail with hash incongruences, but some others fail with this:
- File "x.opam", line 1, characters 0-0:
- Error: Expected formula to be a conjunction of atoms but encountered non-atom
- term 'a | b'
+ Error: Unable to solve dependencies for the following lock directories:
+ Lock directory dune.lock:
+ Can't find all required versions.
+ Selected: x.dev a
+ - a -> (problem)
+ No known implementations at allShouldn't these tests now pass with a proper build?
| let compare_filters filter filter' = | ||
| let get_vars = function | ||
| | OpamTypes.Constraint _ -> [] | ||
| | Filter filter -> Stdlib.List.sort Stdlib.compare (OpamFilter.variables filter) |
There was a problem hiding this comment.
This is basically taken taken from vendor/opam/src/state/opamFileTools.ml as a whole, because I wanted a way to create a Set/Map without having to implement my own compare function.
I assume this is because the ordering of the filters doesn't matter, if the filters are in a different ordering they should still be considered equal.
| in | ||
| match get_vars filter, get_vars filter' with | ||
| | v :: _, v' :: _ -> Stdlib.compare v v' | ||
| | [], _ :: _ -> 1 |
There was a problem hiding this comment.
We don't use numbers for orderings in the dune code, just the constructors of Ordering.
There was a problem hiding this comment.
That's true and I personally also prefer this, but this function gets passed to OpamFormula.compare_formula which requires the integer way of comparing. I can reimplement compare_formula to use Ordering or add a new function to Ordering that would convert comparison functions between the representations. Which of those do you think would be the best approach?
| module Map = Map.Make (T) | ||
| module Set = Set.Make (T) (Map) | ||
|
|
||
| let maximum_package_set formula = |
There was a problem hiding this comment.
The use of this function for non_local_deps isn't correct. Just because a package appears in an or, doesn't mean we're actually using it in the lock directory.
There was a problem hiding this comment.
That is true, but bar of evaluating the formula with all the filters (which would need access to all of the OPAM variables and lead to potentially different solutions on different systems) this is a simple way to determine an overapproximation of the non-local packages. In the worst case this would lead to
- Reporting the non-local package error with non-local package that wouldn't get picked (as we only ever pick one at random for the error message)
- The hash of non-local packages would change if a non-local package changes its dependencies even if we don't actually end up depending on it.
While only an approximation I think the upside is a stable solution across systems. The solver still gets the whole dependency formula and will compute an accurate lock file.
|
@ElectreAAS The failure you quoted makes sense, because the test in The hash changes are an unfortumate side-effect as the non-local dependency set is calculated from a different representation and bar of implementing the OPAM filters into Dune's condition language and then hashing that there is no good way of preserving the previous hashes (or running a hybrid approach of encoding them one way if they can and another if they cannot be represented using dune-lang conditions). I don't think we guarantee that the hashes are stable between versions of Dune. |
The `Dependency_set.t` representation can't deal with disjunctions but in most cases that is not even necessary as the set gets turned into a `filtered_formula` again. Thus it might be easier to keep the original representation and implement the necessary dependency set functionality on top of that. Signed-off-by: Marek Kubica <[email protected]>
Signed-off-by: Marek Kubica <[email protected]>
Signed-off-by: Marek Kubica <[email protected]>
Leonidas-from-XIV
force-pushed
the
filtered-formula-passthrough
branch
from
3183e7e
to
31229fb
Compare
Signed-off-by: Marek Kubica <[email protected]>
Signed-off-by: Marek Kubica <[email protected]>
|
I've replaced the failing test with one that shows that disjunctions in opam files lead to valid lock directory solutions using either side of the disjunction. |
|
Oh nice! Thanks for updating the test |
At the moment when an OPAM package in the local packages we only accept a list of
Atoms, so if it has an disjunction in the dependencies we fail when loading as we attempt to parse the list ofAtoms into aDependency_set.tstructure. This structure only supports a sequence of conjunctions (as that is the only syntax thatdune-projectfiles support at the moment).This PR changes the internal representation of the dependencies to be
OpamTypes.filtered_formulaas when reading OPAM files this is what we get from the OPAM file parser. When we pass the data to the solver it is alsoOpamTpyes.filtered_formulathat we have to encode back to. Thus I suggest keepingOpamTypes.filtered_formulaas dependency specification and compute the information we need from it directly. The information is fairly limited as we only need one example package name for an error message and a hash of the dependencies.The downside of this is that all existing dependency hashes change (see tests), as the representation of the package constraints the hash is calculated over is changing, but I consider this an acceptable compromise.
Fixes #10837