ci(mypy): add mypy check and adjust code for types #439

d33bs · 2024-09-15T19:36:15Z

Description

This PR adds a check through GitHub Actions for mypy in order to make sure Python types align with certain standards within pycytominer. I've also made some adjustments to the code in order to pass mypy checks.

Closes #437

What is the nature of your change?

Bug fix (fixes an issue).
Enhancement (adds functionality).
Breaking change (fix or feature that would cause existing functionality to not work as expected).
This change requires a documentation update.

Checklist

Please ensure that all boxes are checked before indicating that a pull request is ready for review.

I have read the CONTRIBUTING.md guidelines.
My code follows the style guidelines of this project.
I have performed a self-review of my own code.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation.
My changes generate no new warnings.
New and existing unit tests pass locally with my changes.
I have added tests that prove my fix is effective or that my feature works.
I have deleted all non-relevant text in this pull request template.

📚 Documentation preview 📚: https://pycytominer--439.org.readthedocs.build/en/439/

codecov-commenter · 2024-09-15T20:30:14Z

Codecov Report

Attention: Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.73%. Comparing base (c782728) to head (b057fb1).
Report is 36 commits behind head on main.

Files with missing lines	Patch %	Lines
pycytominer/aggregate.py	75.00%	1 Missing ⚠️
pycytominer/cyto_utils/collate.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #439      +/-   ##
==========================================
+ Coverage   94.72%   94.73%   +0.01%     
==========================================
  Files          56       57       +1     
  Lines        3147     3155       +8     
==========================================
+ Hits         2981     2989       +8     
  Misses        166      166

Flag	Coverage Δ
unittests	`94.73% <91.66%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gwaybio

LGTM, but I trust Ken more to give final approval

pycytominer/cyto_utils/output.py

tests/test_cyto_utils/test_output.py

kenibrewer

This is a surprisingly low number of changes for enabling mypy in a repo. This might be because of the ignore_missing_imports and allow_redefinitions config options. What was the change burden like for turning those options off?

d33bs · 2024-09-18T10:38:38Z

Thanks @gwaybio and @kenibrewer for the reviews.

This is a surprisingly low number of changes for enabling mypy in a repo.

I'm not sure, but I wonder if this has to do with the low amount of type hinting alongside compatibility with inferred types. Could be useful to make a more focused effort towards adding type hints throughout.

What was the change burden like for turning those options off?

When I looked here I noticed we no longer needed the allow_redefinitions config option so I've turned this off. ignore_missing_imports results in the following errors / warnings. Generally I turned this on as I felt there might be many dependencies we would have to add and I wasn't certain of the benefit. Maybe this could be an iterative process where we move towards that in another change? I can also see why we might do it here.

pycytominer/__config__.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/output.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/features.py:6: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:6: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:10: error: Skipping analyzing "boto3": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:11: error: Skipping analyzing "botocore": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations.py:13: error: Skipping analyzing "sqlalchemy": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/cell_locations_cmd.py:2: error: Skipping analyzing "fire": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:9: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/operations/transform.py:10: error: Skipping analyzing "scipy.stats": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:11: error: Skipping analyzing "sklearn.base": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/operations/transform.py:12: error: Skipping analyzing "sklearn.preprocessing": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/util.py:8: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/load.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/modz.py:2: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/collate.py:75: error: Skipping analyzing "cytominer_database.ingest": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/collate.py:75: error: Skipping analyzing "cytominer_database": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/collate.py:76: error: Skipping analyzing "cytominer_database.munge": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/cyto_utils/write_gct.py:12: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/DeepProfiler_processing.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: Hint: "python3 -m pip install pandas-stubs"
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: (or run "mypy --install-types" to install all missing stub packages)
pycytominer/cyto_utils/DeepProfiler_processing.py:7: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
pycytominer/aggregate.py:8: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/normalize.py:5: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/normalize.py:6: error: Skipping analyzing "sklearn.preprocessing": module is installed, but missing library stubs or py.typed marker  [import-untyped]
pycytominer/annotate.py:7: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cells.py:4: error: Library stubs not installed for "pandas"  [import-untyped]
pycytominer/cyto_utils/cells.py:21: error: Skipping analyzing "sqlalchemy": module is installed, but missing library stubs or py.typed marker  [import-untyped]
Found 26 errors in 16 files (checked 31 source files)

d33bs · 2024-09-24T14:27:57Z

This is now ready for another look when there's a moment @gwaybio and @kenibrewer . I've added more changes to include type stubs where reasonable. Many packages used by pycytominer do not have their own stubs provided within the package or through external stub packages - these are now specifically ignored via an override setting within the pyproject.toml. I've added additional type stub dependencies to the dev group, as these would now be required to successfully develop and test during development.

Generally I found that it'd be healthy to explore alternative type checkers in the future, especially pyright, where Microsoft has done extensive work to provide custom stubs that likely integrate via python-type-stubs, or pylyzer which would grant speed and possibly other benefits. I didn't feel I could rationalize importing many of the relevant stubs into this project locally - these could be fairly burdensome to manage in addition to their likelihood of changing.

gwaybio

LGTM - let's wait on merging this until we merge #448 (after we merge 448, I am guessing you will need to resolve conflicts and rerun)

kenibrewer

LGTM too! Thanks for looking into the extra imports

gwaybio · 2024-09-27T21:55:54Z

Ready to merge @d33bs !

d33bs · 2024-09-28T12:41:35Z

Thanks @gwaybio and @kenibrewer for your reviews! I double checked from main and it looks like mypy tests still pass. Merging this in.

d33bs force-pushed the add-mypy-checks branch 3 times, most recently from d517035 to 77d1bb4 Compare September 15, 2024 20:24

d33bs force-pushed the add-mypy-checks branch from 77d1bb4 to 7f6ea31 Compare September 16, 2024 13:50

d33bs mentioned this pull request Sep 16, 2024

Increase test coverage for collate #442

Open

ci(mypy): add mypy check and adjust code for types

8478dbd

d33bs force-pushed the add-mypy-checks branch from 7f6ea31 to 8478dbd Compare September 16, 2024 15:27

d33bs requested review from gwaybio and kenibrewer September 16, 2024 17:00

d33bs marked this pull request as ready for review September 16, 2024 17:00

gwaybio reviewed Sep 16, 2024

View reviewed changes

pycytominer/cyto_utils/output.py Show resolved Hide resolved

tests/test_cyto_utils/test_output.py Show resolved Hide resolved

kenibrewer reviewed Sep 17, 2024

View reviewed changes

remove redefinitions config

92b91a1

d33bs added 9 commits September 23, 2024 13:20

update stubs and ignores

f8e7b36

update output for compression typing

90ffa30

update expanduser types for cell_locations

3bf75fb

add ignore for change in types

52e2a4b

adjust for typing

37a5ec7

update celllocations typing

8a363e7

remove ignores

45f1803

add missing library ignore settings; fix aggregate

436e9c4

add test condition for aggregate

b057fb1

d33bs requested review from gwaybio and kenibrewer September 24, 2024 14:28

gwaybio approved these changes Sep 26, 2024

View reviewed changes

kenibrewer approved these changes Sep 26, 2024

View reviewed changes

gwaybio mentioned this pull request Sep 26, 2024

Update error message and docs for features argument to clarify CellProfiler default expectations and how to handle non-CellProfiler data #448

Merged

13 tasks

d33bs merged commit af07d49 into cytomining:main Sep 28, 2024
12 checks passed

d33bs deleted the add-mypy-checks branch September 28, 2024 12:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(mypy): add mypy check and adjust code for types #439

ci(mypy): add mypy check and adjust code for types #439

d33bs commented Sep 15, 2024 •

edited

Loading

codecov-commenter commented Sep 15, 2024 •

edited

Loading

gwaybio left a comment

kenibrewer left a comment

d33bs commented Sep 18, 2024

d33bs commented Sep 24, 2024 •

edited

Loading

gwaybio left a comment

kenibrewer left a comment

gwaybio commented Sep 27, 2024

d33bs commented Sep 28, 2024

ci(mypy): add mypy check and adjust code for types #439

ci(mypy): add mypy check and adjust code for types #439

Conversation

d33bs commented Sep 15, 2024 • edited Loading

Description

What is the nature of your change?

Checklist

codecov-commenter commented Sep 15, 2024 • edited Loading

Codecov Report

gwaybio left a comment

Choose a reason for hiding this comment

kenibrewer left a comment

Choose a reason for hiding this comment

d33bs commented Sep 18, 2024

d33bs commented Sep 24, 2024 • edited Loading

gwaybio left a comment

Choose a reason for hiding this comment

kenibrewer left a comment

Choose a reason for hiding this comment

gwaybio commented Sep 27, 2024

d33bs commented Sep 28, 2024

d33bs commented Sep 15, 2024 •

edited

Loading

codecov-commenter commented Sep 15, 2024 •

edited

Loading

d33bs commented Sep 24, 2024 •

edited

Loading