Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed incorrect "translation invariance" in problem in softmax-regression.md #2592

Closed
wants to merge 1 commit into from
Closed

Conversation

Daniel-Sinkin
Copy link
Contributor

@Daniel-Sinkin Daniel-Sinkin commented Mar 17, 2024

Hello, when working through the exercises inside of softmax-regression.md I've noticed that one of the problem seems to have a small error, namely the function $g$ is not translation invariant, but rather translation equivariant. This means that there should be a $+b$ term outside of the function as well.

Also I've changed the sign of the b to align it closer to what should be done in the last subproblem: "Show that if we choose $b = \mathrm{max}_i x_i$ we end up with a numerically stable implementation."

As such the corrected version of the problem is $g(\underline{x} - b) = g(\underline{x}) - b$.

Proof

$$\begin{aligned} g(\underline{x} - b) &= \log \sum_i \exp(x_i - b) \\\ &= \log \sum_i \exp(x_i) \exp(-b) \\\ &= \log \left( \left( \sum_i \exp(x_i) \right) \exp(-b) \right) \\\ &\overset{1}{=} \log \left(\sum_i \exp(x_i)\right) + \log \exp(-b) \\\ &\overset{2}{=} \log \left(\sum_i \exp(x_i)\right) - b = g(\underline{x}) - b. \end{aligned}$$

Where we have used that (1) $\log(xy) = \log(x) + \log(y)$ and (2) $\log\exp x = x$.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

It's not invariant, but equivariant. The problem is missing the +b outside the function.

Also I've changed the sign of the b to align it closed to what we want to do in the last sub-problem.
@Daniel-Sinkin Daniel-Sinkin closed this by deleting the head repository Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant