Releases: flyteorg/flyte
Flyte v1.11.0-b0 milestone release
Flyte v1.11.0-b0
Beta release to test new idl
Flyte v1.10.7 milestone release
Flyte 1.10.7 Release Notes
We're excited to share the release of Flyte 1.10.7, featuring a broad spectrum of updates, improvements, and bug fixes across the Flyte ecosystem. This release marks a pivotal shift in our development approach, notably with our adoption of buf for protobuf stub generation. This move optimizes our development workflow and discontinues the automatic creation of Java and C++ stubs, making it easier to adapt the generated code for other languages as needed. Additionally, we've upgraded to gRPC-gateway v2, aligning with the latest advancements and recommendations found in the v2 migration guide.
Our sincere gratitude goes to all contributors for their invaluable efforts towards this release.
Core Improvements and Bug Fixes
- Improved error handling for transient secret sync issues, enhancing the robustness of secret management. [PR #4310]
- Introduced Sphinx build for Monodocs, improving documentation generation and integration. [PR #4347]
- Enhanced the Spark plugin by fixing the environment variable
ValueFrom
for pod templates, allowing for more dynamic configurations. [PR #4532] - Optimized fastcache behavior to not cache lookups on node skip, reducing unnecessary cache hits. [PR #4524]
- Removed composition errors from branch nodes, streamlining execution paths. [PR #4528]
- Added support for ignoring warnings related to AWS SageMaker imports, improving integration compatibility. [PR #4540]
- Fixed a bug related to setting the service account from PodTemplate, ensuring correct service account usage. [PR #4536]
- Addressed flaky tests in test_monitor, enhancing test reliability. [PR #4537]
- Updated the boilerplate version and contribution guide, facilitating better community contributions. [PR #4541], [PR #4501]
- Improved documentation build processes by manually creating version files and introducing a conda-lock file for consistent environment setup. [PR #4556], [PR #4553]
- Enhanced array node evaluation frequency optimization by detecting subNode phase updates. [PR #4535]
- Introduced support for failure nodes, allowing workflows to handle failures more gracefully. [PR #4308]
- Made various updates to Go versions, plugin integrations, and GitHub workflows to enhance performance and developer experience. [PR #4534], [PR #4582], [PR #4589]
- Addressed several bugs and made improvements in caching, metadata handling, and task execution, further stabilizing the Flyte platform. [PR #4594], [PR #4590], [PR #4607]
- Streamlined development workflow with the transition to buf for generating protobuf stubs, ceasing the automatic generation of Java and C++ stubs.
- Upgraded to grpc-gateway v2, optimizing API performance and compatibility.
Plugin and Integration Enhancements
- Added new features and fixed bugs in the Spark plugin, Ray Autoscaler integration, and other areas, expanding Flyte's capabilities and integration ecosystem. [PR #4363]
- Updated various dependencies and configurations, ensuring compatibility and security. [PR #4571], [PR #4643]
- Improved the handling and documentation of plugin secrets management, making it easier for users to manage sensitive information. [PR #4732]
Documentation and Community
- Updated community meeting cadence and contribution guidelines, fostering a more engaged and welcoming community. [PR #4699]
- Enhanced documentation through various updates, including the introduction of a new architecture image for FlytePlugins and clarification of propeller scaling. [PR #4661], [PR #4741]
Full Changelog
- Fix transient secret sync error handling by @Tom-Newton in #4310
- Monodocs sphinx build by @cosmicBboy in #4347
- [Spark plugin] Fix environment variable ValueFrom for pod templates by @Tom-Newton in #4532
- fastcache should not cache lookup on node skip by @hamersaw in #4524
- Removed composition error from branch node by @hamersaw in #4528
- ignore warnings related to awssagemaker import by @cosmicBboy in #4540
- [BUG] Fix setting of service_account from PodTemplate by @pvditt in #4536
- Fix flaky test_monitor by @pingsutw in #4537
- Update boilerplate version by @flyte-bot in #4541
- remove hardcoded list of tests by @samhita-alla in #4521
- manually create flytekit/_version.py file in docs build by @cosmicBboy in #4556
- introduce conda-lock file for docs by @cosmicBboy in #4553
- Detect subNode phase updates to reduce evaluation frequency of ArrayNode by @hamersaw in #4535
- Add support failure node by @pingsutw in #4308
- Return InvalidArgument for workflow compilation failures in CreateWorkflow by @katrogan in #4566
- Update to go 1.21 by @eapolinario in #4534
- Update contribution guide by @pingsutw in #4501
- Add flyin plugin to monodocs integrations page by @neverett in #4582
- Use updated cronSchedule in CreateLaunchPlanModel by @pmahindrakar-oss in #4564
- Writing zero length inputs by @hamersaw in #4594
- Feature/add pod pending timeout config by @pvditt in #4590
- Run single-binary gh workflows on all PRs by @eapolinario in #4589
- auto-generate toctree from flytesnacks index.md docs by @cosmicBboy in #4587
- add repo tag and commit associated with the build by @cosmicBboy in #4571
- monodocs - gracefully handle case when external repo doesn't contain tags: use current commit by @cosmicBboy in #4598
- convert commit to string by @cosmicBboy in #4599
- Bug/abort map task subtasks by @pvditt in #4506
- Supporting parallelized workers in ArrayNode subNodes by @hamersaw in #4567
- Don't use experimental readthedocs build.commands config by @cosmicBboy in #4606
- Ignore cache variables by @hamersaw in #4618
- Feature/add cleanup non recoverable pod statuses by @pvditt in #4607
- Agent Metadata Servicer by @Future-Outlier in #4511
- Add Flyin propeller config by @eapolinario in #4610
- Correctly computing ArrayNode maximum attempts and system failures by @hamersaw in #4627
- Agent Sync Plugin by @Future-Outlier in #4107
- Add github token in buf gh action by @eapolinario in #4626
- Update flyte-binary values by @davidmirror-ops in #4604
- Fixing cache overwrite metadata update by @hamersaw in #4617
- Fixing 100 kilobyte max error message size by @hamersaw in #4631
- Add Ray Autoscaler to the Flyte-Ray plugin by @Yicheng-Lu-llll in #4363
- Artifact protos and related changes by @wild-endeavor in #4474
- Remove protoc-gen-validate by @eapolinario in #4643
- Readme update 2023 by @davidmirror-ops in #4549
- Fixing ArrayNode integration with backoff controller by @hamersaw in #4640
- Avoid to use the http.DefaultClient by @andresgomezfrr in #4667
- Update dns policy for sandbox buildkit instance to ClusterFirstWithHo… by @jeevb in #4678
- Updating ArrayNode ExternalResourceInfo ID by @hamersaw in #4677
- Feat: Inject user identity as pod label in K8s plugin by @fg91 in https://githu...
Flyte v1.10.7-b4 milestone release
Flyte v1.10.7-b4 Release
Pre-release testing.
Flyte v1.10.7-b3 milestone release
Flyte v1.10.7-b3 Release
Pre-release testing.
Flyte v1.10.7-b2 milestone release
Flyte v1.10.7-b2 Release
Pre-release testing.
Flyte v1.10.7-b1 milestone release
Flyte v1.10.7-b1 Release
Pre-release testing.
Flyte v1.10.7-b0 milestone release
Flyte v1.10.7-b0 Release
Beta release.
Flyte v1.10.6 milestone release
Flyte 1.10.6 Release
Due to a mishap in the move to the monorepo, we ended up generating the git tags between 1.10.1 to 1.10.5, so in order to decrease the confusion we decided to skip those patch versions and go straight to the next available version.
We've shipped a ton of stuff in this patch release, here are some of the highlights.
GPU Accelerators
You'll be able to get more fine-grained in the use GPU Accelerators in your tasks. Here are some examples:
No preference of GPU accelerator to use:
@task(limits=Resources(gpu="1"))
def my_task() -> None:
...
Schedule on a specific GPU accelerator:
from flytekit.extras.accelerators import T4
@task(
limits=Resources(gpu="1"),
accelerator=T4,
)
def my_task() -> None:
...
Schedule on a Multi-instance GPU (MIG) accelerator with no preference of partition size:
from flytekit.extras.accelerators import A100
@task(
limits=Resources(gpu="1"),
accelerator=A100,
)
def my_task() -> None:
...
Schedule on a Multi-instance GPU (MIG) accelerator with a specific partition size:
from flytekit.extras.accelerators import A100
@task(
limits=Resources(gpu="1"),
accelerator=A100.partition_1g_5gb,
)
def my_task() -> None:
...
Schedule on an unpartitioned Multi-instance GPU (MIG) accelerator:
from flytekit.extras.accelerators import A100
@task(
limits=Resources(gpu="1"),
accelerator=A100.unpartitioned,
)
def my_task() -> None:
...
Improved support for Ray logs
#4266 opens the door for RayJob logs to be persisted.
In #4397 we added support for a link to a Ray dashboard to show up in the task card.
Updated grafana dashboards
We updated the official grafana dashboards in #4382.
Support for Azure AD
A new version of our stow fork added support for Azure AD in flyteorg/stow#9.
Full changelog:
- Restructure Flyte releases by @eapolinario in #4304
- Use debian bookworm as single binary base image by @eapolinario in #4311
- Use local version in single-binary by @eapolinario in #4294
- Accessibility for README by @mishmanners in #4322
- Add tests in
flytepropeller/pkg /controller/executors
from 72.3% to 87.3% coverage by @Future-Outlier in #4276 - fix: remove unused setting in deployment charts by @HeetVekariya in #4252
- Document simplified retry behaviour introduced in #3902 by @fg91 in #4022
- Ray logs persistence by @jeevb in #4266
- Not revisiting task nodes and correctly incrementing parallelism by @hamersaw in #4318
- Fix RunPluginEndToEndTest util by @andresgomezfrr in #4342
- Tune sandbox readiness checks to ensure that sandbox is fully accessi… by @jeevb in #4348
- Chore: Ensure Stalebot doesn't close issues we've not yet triaged. by @brndnblck in #4352
- Do not automatically close stale issues by @eapolinario in #4353
- Fix: Set flyteadmin gRPC port to 80 in ingress if using TLS between load balancer and backend by @fg91 in #3964
- Support Databricks WebAPI 2.1 version and Support
existing_cluster_id
andnew_cluster
options to create a Job by @Future-Outlier in #4361 - Fixing caching on maptasks when using partials by @hamersaw in #4344
- Fix read raw limit by @honnix in #4370
- minor fix to eks-starter.yaml by @guyarad in #4337
- Reporting running if the primary container status is not yet reported by @hamersaw in #4339
- completing retries even if minSuccesses are achieved by @hamersaw in #4338
- Add comment to auth scope by @wild-endeavor in #4341
- Update tests by @eapolinario in #4381
- Update order of cluster resources config to work with both uctl and flytectl by @neverett in #4373
- Update tests in single-binary by @eapolinario in #4383
- Passthrough unique node ID in task execution ID for generating log te… by @jeevb in #4380
- Add Sections in the PR Template by @Future-Outlier in #4367
- Update metadata in ArrayNode TaskExecutionEvents by @hamersaw in #4355
- Fixes list formatting in flytepropeller arch docs by @thomasjpfan in #4345
- Update boilerplate end2end tests by @hamersaw in #4393
- Handle all ray job statuses by @EngHabu in #4389
- Relocate sandbox config by @davidmirror-ops in #4385
- Refactor task logs framework by @jeevb in #4396
- Add support for displaying the Ray dashboard when a RayJob is active by @jeevb in #4397
- Disable path filtering for monorepo components by @eapolinario in #4404
- Silence NotFound when get task resource by @honnix in #4388
- adding consoleUrl parameterization based on partition by @lauralindy in #4375
- [Docs] Sensor Agent Doc by @Future-Outlier in #4195
- [flytepropeller] Add Tests in v1alpha.go including
array_test.go
,branch_test.go
,error_test.go
, andiface_test.go
with 0.13% Coverage Improvement by @Future-Outlier in #4234 - Add more context for ray log template links by @jeevb in #4416
- Add ClusterRole config for Ray by @davidmirror-ops in #4405
- Fix and update scripts for generating grafana dashboards by @Tom-Newton in #4382
- Add artifacts branch to publish to buf on push by @squiishyy in #4450
- Add service monitor for flyte admin and propeller service by @vraiyaninv in #4427
- Fix Kubeflow TF Operator
GetTaskPhase
Bug by @Future-Outlier in #4469 - Instrument opentelemetry by @hamersaw in #4357
- Delete the .github folder from each subdirectory by @pingsutw in #4480
- Fix the loop variable scheduler issue by @pmahindrakar-oss in #4468
- Databricks Plugin Setup Doc Enhancement by @Future-Outlier in #4445
- Put ticker back in place in propeller gc by @eapolinario in #4490
- Store failed execution in flyteadmin by @iaroslav-ciupin in #4390
- Moving from flyteadmin - Upgrade coreos/go-oidc to v3 to pickup claims parsing fixes by @eapolinario in #4139
- Bump flyteorg/stow to 0.3.8 by @eapolinario in #4312
- Remove 'needs' from generate_flyte_manifest by @eapolinario in #4495
- Update Flyte components by @flyte-bot in #4302
- Modify how flytecopilot version is parsed from values file by @eapolinario in #4496
- Ignore component tags in goreleaser by @eapolinario in #4497
- Fix indentation of
shell: task
by @eapolinario in #4498 - Implemented simple echo plugin for testing by @hamersaw in #4489
- Correctly handle resource overrides in KF plugins by @jeevb in #4467
- Remove deprecated InjectDecoder by @EngHabu in #4507
- Fix $HOME resolution and webhook namespace by @EngHabu in #4509
- Add note on updating sandbox cluster configuration by @jeevb in #4510
- Add New PR Template by @Future-Outlier in #4512
- [Docs] Databricks Agent Doc by @Future-Outlier in #4008
- Bump version of goreleaser gh action to v5 by @eapolinario in #4519
- Kf operators use
GetReplicaFunc
(Error Handling) by @Future-Outlier in #4471
New Contributors
- @HeetVekariya made their first contribution in #4252
- @andresgomezfrr made their first contribution in #4342
- @brndnblck made their first contribution in #4352
- @guyarad made their first contribution in #4337
- @neverett made their first contribution in #4373
- @thomasjpfan made their first...
Flyte v1.10.6-b0 milestone release
Flyte v1.10.6-b0 Release
Beta release.
Flyte v1.10.0 milestone release
Flyte v1.10.0 Release
This release marks the move to the (monorepo](#4014). From now on the development of backend components is going to be done in the Flyte main repo.
Flyte Agents are also getting a major lift. They are not only more performant, but we're also announcing support for a variety of agents, including Airflow, Memverge, Snowflake, Databricks.
Programmatically consuming inputs and outputs using flyteremote became a lot easier with the introduction of code snippets directly in the UI. For example, notice the small blurbs below inputs and outputs in the side panel:
You'll now be able to use offloaded types in eager workflows.
More ergonomic improvements to pyflyte, including the inclusion of a progress bar, the ability to activate launchplans, and the ability to interact with gate nodes in local executions.
And much more. Here's the exhaustive list of changes:
Flytekit
- Better error messaging for overrides by @kumare3 in flyteorg/flytekit#1807
- Run remote Launchplan from
pyflyte run
by @kumare3 in flyteorg/flytekit#1785 - Add is none function by @pingsutw in flyteorg/flytekit#1757
- Dynamic workflow should not throw nested task warning by @oliverhu in flyteorg/flytekit#1812
- Add a manual image building GH action by @wild-endeavor in flyteorg/flytekit#1816
- Enable Azure Workload Identity for
fsspec
inflytekit
by @fiedlerNr9 in flyteorg/flytekit#1813 - Fix list of annotated structured dataset by @wild-endeavor in flyteorg/flytekit#1817
- Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit by @chaohengstudent in flyteorg/flytekit#1819
- Async agent delete function for while loop case by @Future-Outlier in flyteorg/flytekit#1802
- fix docs warnings by @samhita-alla in flyteorg/flytekit#1827
- Fix extract_task_module by @pingsutw in flyteorg/flytekit#1829
- Feat: Add type support for pydantic BaseModels by @ArthurBook in flyteorg/flytekit#1660
- Make FlyteRemote example slightly more copy/pastable by @katrogan in flyteorg/flytekit#1830
- Pyflyte meta inputs by @kumare3 in flyteorg/flytekit#1823
- Use mashumaro to serialize/deserialize dataclass by @hhcs9527 in flyteorg/flytekit#1735
- Databricks Agent by @Future-Outlier in flyteorg/flytekit#1797
- Prometheus metrics by @pingsutw in flyteorg/flytekit#1815
- Pyflyte register optionally activates schedule by @kumare3 in flyteorg/flytekit#1832
- Remove versions 3.9 and 3.10 by @wild-endeavor in flyteorg/flytekit#1831
- Snowflake agent by @hhcs9527 in flyteorg/flytekit#1799
- Update agent metric name by @pingsutw in flyteorg/flytekit#1835
- MemVerge MMCloud Agent by @edwinyyyu in flyteorg/flytekit#1821
- Add download badges to the readme by @pingsutw in flyteorg/flytekit#1836
- Eager local entrypoint and support for offloaded types by @cosmicBboy in flyteorg/flytekit#1833
- update requirements and add snowflake agent to api reference by @samhita-alla in flyteorg/flytekit#1838
- Fix: Make sure decks created in elastic task workers are transferred to parent process by @fg91 in flyteorg/flytekit#1837
- add accept grpc by @wild-endeavor in flyteorg/flytekit#1841
- Feat: Enable
flytekit
to authenticate with proxy in front of FlyteAdmin by @fg91 in flyteorg/flytekit#1787 - Backfill command now supports failure-policy by @kumare3 in flyteorg/flytekit#1840
- Pass cluster pool when creating executions by @iaroslav-ciupin in flyteorg/flytekit#1208
- Add more clear error message when fetching secrets by @ysysys3074 in flyteorg/flytekit#1847
- Pyflyte run workflows correctly handles Optional[TYPE] = None by @cosmicBboy in flyteorg/flytekit#1849
- Bump gitpython from 3.1.32 to 3.1.35 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in flyteorg/flytekit#1828
- Bump cryptography from 41.0.3 to 41.0.4 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in flyteorg/flytekit#1844
- update codecov config in pythonbuild ci by @cosmicBboy in flyteorg/flytekit#1852
- Improved gate node handling by @kumare3 in flyteorg/flytekit#1850
- update codecov yaml, make eager wf test more stable by @cosmicBboy in flyteorg/flytekit#1854
- more codecov updates by @cosmicBboy in flyteorg/flytekit#1856
- make mlflow plugin work with python 3.11 by @bcvanmeurs in flyteorg/flytekit#1855
- Agent get secret function by @Future-Outlier in flyteorg/flytekit#1851
- Better Azure blob storage support by @Tom-Newton in flyteorg/flytekit#1842
- Remove Envd from dockerfile by @pingsutw in flyteorg/flytekit#1814
- Update agent uri format by @hhcs9527 in flyteorg/flytekit#1861
- modified setup.py in flytekit-mmcloud by @helenzhangyc in flyteorg/flytekit#1864
- Label request_failure metric with error_code by @pingsutw in flyteorg/flytekit#1862
- add back Edwin's name by @helenzhangyc in flyteorg/flytekit#1868
- Fix py38 aync agent service and add async agent test by @Future-Outlier in flyteorg/flytekit#1866
- Pin because not installing by @wild-endeavor in flyteorg/flytekit#1872
- Bump pillow from 10.0.0 to 10.0.1 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in flyteorg/flytekit#1867
- Prune hosted github runner to free up space for sandbox by @jeevb in flyteorg/flytekit#1875
- Fixes Auth issue, where refresh token is expired by @kumare3 in flyteorg/flytekit#1873
- Name field is incorrect assigned by @kumare3 in flyteorg/flytekit#1874
- [Docs]Updated examples for customizing resources by @LunarMarathon in flyteorg/flytekit#1871
- change key path to SNOWFLAKE_PRIVATE_KEY by @hhcs9527 in flyteorg/flytekit#1879
- Run notebook task on aws batch by @pingsutw in flyteorg/flytekit#1859
- Automatically Convert
name
Argument andregistry
Argument to Lowercase in ImageSpec by @Future-Outlier in flyteorg/flytekit#1881 - Dep: Pin min required flytekit version in IAP plugin by @fg91 in flyteorg/flytekit#1878
- Remove duplicate click option by @wild-endeavor in flyteorg/flytekit#1883
- Bump urllib3 from 1.26.16 to 1.26.17 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in flyteorg/flytekit#1865
- Improve error message in ListTransformer when not passing a list by @fg91 in flyteorg/flytekit#1882
- Update
typing_extension
Any import to typing due totyping_extension
version compatibility by @hfurkanvural in flyteorg/flytekit#1877 - Bump gitpython from 3.1.35 to 3.1.37 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in flyteorg/flytekit#1885
- Airflow agent by @pingsutw in flyteorg/flytekit#1725
- Add mmcloud plugin into the agent image by @pingsutw in flyteorg/flytekit#1891
- Fix map_task sensitive to argument order by @chaohengstudent in flyteorg/flytekit#1888
- Add enable_deck to task decorator by @eapolinario in flyteorg/flytekit#1898
- Add image transformer by @pingsutw in flyteorg/flytekit#1901
- add default project domain in execute launch plan by @troychiu in flyteorg/flytekit#1894
- Add Databricks and Snowflake agent to Dockerfile by @Future-Outlier in flyteorg/flytekit#1905
- Fix module loading bug in instance tracker class by @cosmicBboy in flyteorg/flytekit#1902
Flyteadmin
- Fixed bug for incorrect name (#4175)
- Add version control for ArrayNode eventing (#4165)
- Update golang experimental dep (#4177)
- Add Additional gRPC service handler hook (#4171)
- Handle batched TaskExecutionEvent reasons (#615)
- Lint flyteadmin/auth/handlers_test.go
- #patch Update FromWorkflowModel API (#617)
- add codespell: config, workflow and have typos fixed (#4054)
- Handle batched TaskExecutionEvent reasons (#615)
- Update FromWorkflowModel API (#617)
- Update boilerplate version (#605)
- Reduce a warning to info (#612)
- Logout hook plugin (#611)
- Expire flyte_idt cookie at logout (#610)
- Use configured domain on expired cookie...