Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Inline SQL queries using boolean parameters #15525

Merged
merged 39 commits into from
Jul 26, 2023
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
5cfdae4
Fixing SQL Query
hi-anshul May 3, 2023
cd4359a
Inline SQL queries using boolean parameters #15515
hi-anshul May 3, 2023
3857478
Update purge_events.py
hi-anshul May 3, 2023
2ea3b58
Merge branch 'develop' into develop
hi-anshul May 3, 2023
44aedd8
Merge branch 'matrix-org:develop' into develop
hi-anshul May 22, 2023
7d60307
Updated all Inline SQL queries using boolean parameters as supported …
hi-anshul May 22, 2023
36c4012
Updated all Inline SQL queries using boolean parameters as supported …
hi-anshul May 22, 2023
e5c5d35
Merge branch 'develop' into develop
hi-anshul May 22, 2023
07e90d4
Merge branch 'matrix-org:develop' into develop
hi-anshul Jun 16, 2023
c29c05f
Merge branch 'matrix-org:develop' into develop
hi-anshul Jun 19, 2023
be3fbc1
Merge branch 'matrix-org:develop' into develop
hi-anshul Jun 19, 2023
8737a24
Updated Inline SQL queries using boolean parameters as supported in S…
hi-anshul Jun 19, 2023
24f4de0
Merge branch 'develop' into develop
hi-anshul Jun 20, 2023
7eb4f91
Update 15515.misc
hi-anshul Jun 21, 2023
e5a668d
Merge branch 'matrix-org:develop' into develop
hi-anshul Jun 21, 2023
aa97aa2
Merge branch 'develop' into develop
hi-anshul Jun 21, 2023
f733e25
Update event_federation.py
hi-anshul Jun 23, 2023
7ae7fef
Update stream.py
hi-anshul Jun 23, 2023
b33a4d2
Merge branch 'matrix-org:develop' into develop
hi-anshul Jun 23, 2023
152e015
Merge branch 'matrix-org:develop' into develop
hi-anshul Jul 5, 2023
97f337b
Updated Inline SQL queries using boolean parameters as supported in S…
hi-anshul Jul 5, 2023
53d04b9
Merge branch 'develop' into develop
hi-anshul Jul 5, 2023
19e15c3
Update changelog.d/15515.misc
MadLittleMods Jul 5, 2023
32d8dd4
Rename 15515.misc to 15525.misc
hi-anshul Jul 6, 2023
99e265b
Merge branch 'develop' into develop
hi-anshul Jul 6, 2023
e871524
Update stream.py
hi-anshul Jul 6, 2023
1c5a66e
Merge branch 'matrix-org:develop' into develop
hi-anshul Jul 6, 2023
3e7c901
Update synapse/storage/databases/main/purge_events.py
hi-anshul Jul 7, 2023
34e98ec
Merge branch 'develop' into develop
hi-anshul Jul 7, 2023
66b8ab0
Update synapse/storage/databases/main/stream.py
hi-anshul Jul 7, 2023
634b269
Merge branch 'develop' into develop
hi-anshul Jul 8, 2023
8f7029d
Merge branch 'develop' into develop
hi-anshul Jul 11, 2023
ea1fbfc
Update synapse/storage/databases/main/events.py
hi-anshul Jul 12, 2023
cafd08e
Merge branch 'matrix-org:develop' into develop
hi-anshul Jul 23, 2023
aaa1ad2
Updated push_rule.py
hi-anshul Jul 23, 2023
c4340ff
Merge branch 'develop' into develop
clokep Jul 24, 2023
37d3013
Merge branch 'matrix-org:develop' into develop
hi-anshul Jul 26, 2023
b9e505a
Update room.py
hi-anshul Jul 26, 2023
4506e7f
Merge branch 'develop' into develop
hi-anshul Jul 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/15515.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Updated all Inline SQL queries using boolean parameters as supported in sqlite 3.27
MadLittleMods marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 2 additions & 2 deletions synapse/storage/databases/main/events.py
Original file line number Diff line number Diff line change
Expand Up @@ -1455,8 +1455,8 @@ def _update_outliers_txn(
},
)

sql = "UPDATE events SET outlier = ? WHERE event_id = ?"
txn.execute(sql, (False, event.event_id))
sql = "UPDATE events SET outlier = False WHERE event_id = ?"
Copy link
Contributor

@MadLittleMods MadLittleMods May 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've updated most of the queries, I hardly found any other than these. There're not much SQL queries that needs to be inline.

I think there are a lot more of these instances. If I just search for outlier = ?, I see 5 things and only 3 places changed in this PR.

We probably need to go through any boolean column in synapse/storage/schema/main/full_schemas/72/full.sql.postgres and do the same search across the codebase.

And figure out a plan for how you want to tackle and split things if there are too many.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need to go through any boolean column in synapse/storage/schema/main/full_schemas/72/full.sql.postgres and do the same search across the codebase.

I think that would be a good way to check all the inline queries in the codebase.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Note that the schema dump is a little out of date; there will likely be new boolean columns added since then.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the synapse_port_db script might have the most up-to-date list of boolean columns to check:

BOOLEAN_COLUMNS = {
"access_tokens": ["used"],
"account_validity": ["email_sent"],
"device_lists_changes_in_room": ["converted_to_destinations"],
"device_lists_outbound_pokes": ["sent"],
"devices": ["hidden"],
"e2e_fallback_keys_json": ["used"],
"e2e_room_keys": ["is_verified"],
"event_edges": ["is_state"],
"events": ["processed", "outlier", "contains_url"],
"local_media_repository": ["safe_from_quarantine"],
"presence_list": ["accepted"],
"presence_stream": ["currently_active"],
"public_room_list_stream": ["visibility"],
"pushers": ["enabled"],
"redactions": ["have_censored"],
"room_stats_state": ["is_federatable"],
"rooms": ["is_public", "has_auth_chain_index"],
"users": ["shadow_banned", "approved"],
"un_partial_stated_event_stream": ["rejection_status_changed"],
"users_who_share_rooms": ["share_private"],
"per_user_experimental_features": ["enabled"],
}

txn.execute(sql, (event.event_id))
hi-anshul marked this conversation as resolved.
Show resolved Hide resolved

# Update the event_backward_extremities table now that this
# event isn't an outlier any more.
Expand Down
11 changes: 5 additions & 6 deletions synapse/storage/databases/main/purge_events.py
Original file line number Diff line number Diff line change
Expand Up @@ -249,12 +249,11 @@ def _purge_history_txn(
# Mark all state and own events as outliers
logger.info("[purge] marking remaining events as outliers")
txn.execute(
"UPDATE events SET outlier = ?"
" WHERE event_id IN ("
" SELECT event_id FROM events_to_purge "
" WHERE NOT should_delete"
")",
(True,),
"UPDATE events SET outlier = True"
"WHERE event_id IN ("
hi-anshul marked this conversation as resolved.
Show resolved Hide resolved
" SELECT event_id FROM events_to_purge "
" WHERE NOT should_delete"
")"
)

# synapse tries to take out an exclusive lock on room_depth whenever it
Expand Down
4 changes: 2 additions & 2 deletions synapse/storage/databases/main/registration.py
Original file line number Diff line number Diff line change
Expand Up @@ -454,9 +454,9 @@ def select_users_txn(
) -> List[Tuple[str, int]]:
sql = (
"SELECT user_id, expiration_ts_ms FROM account_validity"
" WHERE email_sent = ? AND (expiration_ts_ms - ?) <= ?"
" WHERE email_sent = False AND (expiration_ts_ms - ?) <= ?"
)
values = [False, now_ms, renew_at]
values = [now_ms, renew_at]
txn.execute(sql, values)
return cast(List[Tuple[str, int]], txn.fetchall())

Expand Down