Visualizing Filters, CNN, Can magnitude of unit vector (X) be approximately greater than or equal to one?


Given:
h_{14} = w_{1}x_{1}+w_{2}x_{2}+w_{3}x_{3}+w_{4}x_{4} is the dot product of the vectors
W = [w_{1}, w_{2}, w_{3}, w_{4}] and X = [x_{1}, x_{2}, x_{3}, x_{4}] written as W^{T}X. The neuron h_{14} will fire maximally when X = \frac{W}{||W||}:

If X is a unit vector defined by \frac{W}{||W||} then how is X = W same as X = \frac{W}{||W||} if magnitude of unit vector is not exactly equal to 1?

Say for example, X = [0.4, 0.5, 0.6, 0.4] and W = [0.4, 0.5, 0.6, 0.4].

Considering two values for X after the decimal point:
Norm of W divided with respect to W yields X as [0.43, 0.53, 0.64, 0.43] and the magnitude results in 1.0297. Now, this is clearly greater than 1 and therefore can X be considered a unit vector here?

Considering one value for X after the decimal point:
Norm of W divided with respect to W yields X as [0.4, 0.5, 0.6, 0.4] and the magnitude results in 0.964. Now, in this case the magnitude is less than 1 but can 0.964 be approximated as 1 and thus can this approximation scenario be considered always? Is this the typical case a machine approximates?

1 Like

I think there was a small mistake in the slide. The quote should have been:

The neuron h_{14} will fire maximally when X \propto \frac{W}{||W||}

Or rather:

The neuron h_{14} will fire maximally when \hat{X } = \frac{W}{||W||}

Which means that, the dot product of W.X will have the highest value only when the vectors \vec{w} and \vec{x} are in the same direction.

Why? Note that cos \theta (from dot product formula) is highest only when \theta = 2n \pi .

Please let me know if this does not answer your question. :slight_smile:

2 Likes

Yes. Thanks for the correction in notation. Now it makes sense as to how vectors X and W are the same.

1 Like

Added CNN to the subject line, so that this nice discussion appears when searching for CNN.

1 Like