Improve CTMS updates from preference center

[jwhitlock]

During testing in basket-dev, not all changes in the preference center were reflected in CTMS. When a user updates subscriptions in preference center, the updates should be reflected in CTMS:

• Changing the email format should change the email format of existing subscriptions
• Changing the email language should change the email language of existing subscriptions
• The subscribed list should be the same as the one in SFDC

Currently, CTMS is copying the updates sent to SFDC. It appears that these updates are minimized (by bedrock or basket) to only send the changes. We may need to proactively copy more data from SFDC to CTMS to ensure that CTMS is fully synced with SFDC by the end of the update.

It is also possible that there is a bug in Bedrock's interface to Basket, and fixes will be needed in that code base.

[jwhitlock]

I think it is also possible to add the unsubscribe reason, even though it is done in two requests:

1. In the first request, mark the newsletter as unsubscribed ("subscribed": false)
2. In the second request, populate the reason for unsubscribed newsletters without reasons ("unsub_reason": null)

This would require reading the full CTMS newsletter records into the basket format (suggested key newsletter_details), and selectively updating those records.

[pmac]

Unsubscribe reason only really needs to be a contact level attribute as it's 1:1 with an email. They are only asked if they unsubscribe from everything by checking the box for the global unsubscribe. So it's not really associated with any one newsletter. They don't get the prompt if they just unsub from any individual one.

[pmac]

Also, just for clarity, what basket gets for newsletters from the pref center form post is the list of newsletters to which the user would like to be subscribed, so to figure out which they've unchecked basket compares the list of newsletters it receives with the list to which the user was subscribed. That's the only way basket has of knowing exactly what's happened. Then it can update the subscriptions based on what the user requested. So you're probably right that SFDC and CTMS can get out of sync if CTMS is relying on SFDC state when deciding what changes to make. We could handle these separately in basket to ensure that the end states are the same.

[jwhitlock]

Unsubscribe reason only really needs to be a contact level attribute as it's 1:1 with an email. They are only asked if they unsubscribe from everything by checking the box for the global unsubscribe. So it's not really associated with any one newsletter. They don't get the prompt if they just unsub from any individual one.

True, this is probably my fault during the CTMS API implementation, when I was designing per-newsletter stats. From non-Github discussions, it sounds like the source URL is most useful as an email-level parameter, set once per email, rather than a per-newsletter parameter like CTMS. If it was earlier in the project, I'd propose simplifying the CTMS data model, but I think it is too complex to coordinate between the services (Basket, CTMS, MoFo, Acoustic) before the end of the month.

Also, just for clarity, what basket gets for newsletters from the pref center form post is the list of newsletters to which the user would like to be subscribed, so to figure out which they've unchecked basket compares the list of newsletters it receives with the list to which the user was subscribed. That's the only way basket has of knowing exactly what's happened. Then it can update the subscriptions based on what the user requested. So you're probably right that SFDC and CTMS can get out of sync if CTMS is relying on SFDC state when deciding what changes to make. We could handle these separately in basket to ensure that the end states are the same.

We have a shrinking time window when we'll have both CTMS and SFDC. Soon, Basket will only be getting data from CTMS, and any mismatch bugs will go away (and become lost data bugs). I'm not confident I'm going to have enough time to fix things during the CTMS / SFDC window.

[pmac]

From non-Github discussions, it sounds like the source URL is most useful as an email-level parameter, set once per email, rather than a per-newsletter parameter like CTMS.

I believe the opposite is true. In my discussions with various people in Marketing they've always wanted to be able to store the source URL per subscription. In fact we were doing that in a separate table (Data Extension) in SFMC since we couldn't do it with the data model in SFDC.

[pmac]

The unsubscribe reason is per person, but source url, language, format, and join date are all things we should track per subscription.