Move to pip-compile-multi #786
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Switch to pip-compile-multi, splitting out dev deps from prod deps It's worked fine, albeit with a TODO in the Dockerfile to actually build a separate dev-specific image, which will come in a separate commit. Went with the pattern of auto-installing the dependency-management tool via the makefile. Will likely switch this to an alert + install instructions This change was needed because in-container compilation was failing and various pip+pip-tools versions attempted clashed due to ``` TypeError: make_requirement_preparer() got an unexpected keyword argument 'wheel_download_dir' ``` which may have been a chicken-and-egg problem, but was not worth solving when a simpler route was around. * Rebuild requirements on the host, not in the container, but retain Makefile commands as the primary invocation route - `make compile-requirements` and `make upgrade-requirements`
…mpile-multi Now, instead of force-intalling pip-compile-multi, we get a prompt showing us what do to (and so providing some flexibility) Also note that some dev subdeps have been minorly bumped. IMPORTANT: if you currently try to recompile deps results in a failure with long-unmaintained suds-jurko, so this will need swapping out. The current set of deps builds and passes tests, at least
No longer needed and was causing a dependency install issue due to its suds-jurko subdep
Invoke with: $ make check-requirements and update hard-pinned dependencies as you wish
robhudson
previously approved these changes
Jan 24, 2022
|
Hmm, this failed for me b/c I don't have mysql on my host machine: |
This gives us guaranteed install stability regardless of the source platform being used to run the update Note that pip-tools and pip-compile-multi are installed ONLY for compiling or upgrading requirements, rather than adding them to the main build. pip-tools needed pinning to a recent version to get around incompatibility issues with pip 21 and pip-tools.
|
@robhudson @pmac One additional point is that The executable in question seems to be so, adding that in now and trying it out UPDATE: installing |
stevejalim
dismissed
robhudson’s stale review
Refactored, so re-requesting review
stevejalim
force-pushed
the
782-move-to-pip-compile-multi
branch
from
c839743
to
b556556
Compare
stevejalim
force-pushed
the
782-move-to-pip-compile-multi
branch
from
b556556
to
06833cb
Compare
One can still compile the docs by using the Makefile in /docs and issuing $ make clean html
7 tasks
robhudson
reviewed
Jan 31, 2022
* hard-pin latest working combo of pip, pip-tools and pip-compile-multi * add in custom header to make recompliation step unambiguous * update Makefile: remove unneeded upgrade-requirements option; update help * recompile reqs with minor bumps
pmac
reviewed
Mar 1, 2022
| # -r requirements/prod.in | ||
| # pytz-deprecation-shim | ||
| # tzlocal | ||
| billiard==3.6.4.0 \ |
There was a problem hiding this comment.
We should just double check that this version is compatible with our version of Celery.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changeset swaps Basket form pip-tools to using pip-compile-multi as our chosen dependency helper, plus retains and evolves a Makefile-driven approach to using pip-compile-multi.
Notes:
UPDATED: this continues to compile requirements inside the Docker container
for now, while we generate separate
requirements/prod.txtandrequirements/dev.txtfiles, we continue to use the superset of dependencies (requirements/dev.txt) for building the single Docker image. Splitting Docker images is deferred to Refactor Docker setup to support separate dev/test and prod images, similar to Bedrock #784 to keep this change cleaner.one dependency - the FuelSDK for Salesforce - had to be removed as it has a very stale dependency and patching the supply chain isn't worth it as FuelSDK isn't used in the Basket codebase any more, it appears.
To test
make compile-requirementsmake compile-requirementsagain and confirm that the deps are rebuilt, but have no changesmake clean build testto ensure that full images still build