Intuition for systems of equations

Intuitively, we can think of a system of equations as a set of requests. Let’s imagine to have a group of people in front of us, and to have to give a task to each of them. An informal example system could be the following:

•  Anna, solve a system of linear equations;
• George, go to the beach and have fun;
• Luke, prevent Anna from ringing social services.

In this form, a solution to the system consists in a list of pairings person-task that satisfies the demands detailed above. In other words, giving a solution to the system amounts to saying what Anna should do, what George should do, and what Luke should do, so that the demands are satisfied. In the example above, Anna should solve a system of equations, George should go to the beach, and Luke should prevent Anna from ringing social services.

It seems pretty obvious, but this intuition will be useful when covering over/underdetermined systems.

Overdetermined systems of equations

Let’s think of having to give orders to a large number of people. It might happen that, when getting to the last person, we forget to whom we have already given an order and to whom not, and we might end up repeating some orders:

• Anna, do the laundry;
• George, go to the beach;
• Luke, get Anna’s laundry dirty;
• Sophie, prevent Luke from dirtying the laundry;
• …
• George, go to the beach.

Here George has received his order twice. In these cases, we say that the system is overdetermined, because it has more orders than people. The example above is innocuous, because George is simply told to the same thing twice. The simplest mathematical example of such a system is when two equations are proportional to each other:

This is an overdetermined system with a solution: . The second equation is just redundant, like a game in which the second rule states to follow the first.

How about the following instead:

• Anna, do the laundry;
• George, go to the beach;
• Luke, get Anna’s laundry dirty;
• Sophie, prevent Luke from dirtying the laundry;
• …
• George, bake a cake.

Here George gets two clashing orders, and is rightfully confused: he cannot go to the beach and bake a cake at the same time. He is going to disappoint us no matter what. Indeed, this system is not only overdetermined, because there are more orders than people, but is also without solution. In fact, we are unable to come up with a list of pairings people-tasks as before. If George would go to the beach, he would be ignoring the baking order; if he would bake a cake, he would be ignoring the beach order. There is no way out: there is no solution! It is a bit like a game where the second rule says not to follow the first: it is impossible to play a game like that!

The simplest mathematical example of such a system is:

which does not have a solution because we ask to be and at the same time — a bit like asking your neighbor to be male and female at the same time (but not queer).

So once again: when a system of equations has more equations than unknowns, we say it is overdetermined. It means that too many rules are being imposed at once, and some of them may be conflicting. However, it is false to state that an overdetermined system does not have any solution: it may or it may not. If the surplus commands are just reformulations of other orders, then it is not a problem: the system does have a solution.

Underdetermined systems of equations

If we give less orders than the number of people, we say the system is underdetermined. When this happens, at least one person must have not received any command. This time, the idea is that people who do not receive any command are free to do whatever they want.

For example, let’s imagine again to have Anna, George, and Luke lined in front of us. If our commands are:

• Anna, do the laundry;
• George, go to the beach.

then Luke has not received any order. Maybe he will go to the park, maybe he will prevent Anna from ringing social services… he is free to do whatever he wants: the options are infinite! In these cases, we say that Luke is a free variable. As long as Anna and George stick to what they are told, each of Luke’s options makes for a solution: that is why the system has an infinite number of solutions.

As a mathematical example, think of being asked to find values for satisfying the following system:

great, but what about ? Here is a free variable and solutions are infinite.

However, there can also be undetermined systems with no solution. That is the case when we give too few orders, and some of them are conflicting with each other. Again with our favorite trio:

• Anna, do the laundry;
• George, go to the beach;
• Anna, go to the park.

Not only are we not saying anything to Luke here, but we are also giving clashing orders to Anna. So even if the absence of commands to Luke would allow infinite solutions, Anna’s impossibility to satisfy the constraints makes it so that no solution exists.

Examples and final remarks

All in all, there are no strict rules. What appears to be an overdetermined system could turn out to be an underdetermined one, and an underdetermined system could have no solution.

Finally, notice that in mathematical reality commands usually address more than one person at a time. A system of equations in real life is something like:

Here the intuition gets trickier, because each command mixes at least two people, and is hard to render in natural language. Still, the orders analogy is useful in understanding what underdetermined and overdetermined systems are and why they have infinite or no solutions.

    in
An apparently overdetermined system which is actually underdetermined and does not even have a solution.

    in
An overdetermined which does not have a solution.

The post Overdetermined and underdetermined systems of equations put simply appeared first on Quick Math Intuitions.

Orthogonal projection: how to build a projector

Case 1 – 2D projection over (1,0)

It is quite straightforward to understand that orthogonal projection over (1,0) can be practically achieved by zeroing out the second component of any 2D vector, at last if the vector is expressed with respect to the canonical basis . Albeit an idiotic statement, it is worth restating: the orthogonal projection of a 2D vector amounts to its first component alone.

How can this be put math-wise? Since we know that the dot product evaluates the similarity between two vectors, we can use that to extract the first component of a vector . Once we have the magnitude of the first component, we only need to multiply that by itself, to know how much in the direction of we need to go. For example, starting from , first we get the first component as ; then we multiply this value by e_1 itself: . This is in fact the orthogonal projection of the original vector. Writing down the operations we did in sequence, with proper transposing, we get

One simple and yet useful fact is that when we project a vector, its norm must not increase. This should be intuitive: the projection process either takes information away from a vector (as in the case above), or rephrases what is already there. In any way, it certainly does not add any. We may rephrase our opening fact with the following proposition:

PROP 1:

This is can easily be seen through the pitagorean theorem (and in fact only holds for orthogonal projection, not oblique):

Case 2 – 2D projection over (1,1)

Attempt to apply the same technique with a random projection target, however, does not seem to work. Suppose we want to project over . Repeating what we did above for a test vector , we would get

This violates the previously discovered fact the norm of the projection should be than the original norm, so it must be wrong. In fact, visual inspection reveals that the correct orthogonal projection of is .

The caveat here is that the vector onto which we project must have norm 1. This is vital every time we care about the direction of something, but not its magnitude, such as in this case. Normalizing yields . Projecting over is obtained through

which now is indeed correct!

PROP 2: The vector on which we project must be a unit vector (i.e. a norm 1 vector).

Case3 – 3D projection on a plane

A good thing to think about is what happens when we want to project on more than one vector. For example, what happens if we project a point in 3D space onto a plane? The ideas is pretty much the same, and the technicalities amount to stacking in a matrix the vectors that span the place onto which to project.

Suppose we want to project the vector onto the place spanned by . The steps are the same: we still need to know how much similar is with respect to the other two individual vectors, and then to magnify those similarities in the respective directions.

The only difference with the previous cases being that vectors onto which to project are put together in matrix form, in a shape in which the operations we end up making are the same as we did for the single vector cases.

The rise of the projector

As we have seen, the projection of a vector over a set of orthonormal vectors is obtained as

And up to now, we have always done first the last product , taking advantage of associativity. It should come as no surprise that we can also do it the other way around: first and then afterwards multiply the result by . This makes up the projection matrix. However, the idea is much more understandable when written in this expanded form, as it shows the process which leads to the projector.

THOREM 1: The projection of over an orthonormal basis is

So here it is: take any basis of whatever linear space, make it orthonormal, stack it in a matrix, multiply it by itself transposed, and you get a matrix whose action will be to drop any vector from any higher dimensional space onto itself. Neat.

Projector matrix properties

• The norm of the projected vector is less than or equal to the norm of the original vector.
• A projection matrix is idempotent: once projected, further projections don’t do anything else. This, in fact, is the only requirement that defined a projector. The other fundamental property we had asked during the previous example, i.e. that the projection basis is orthonormal, is a consequence of this. This is the definition you find in textbooks: that . However, if the projection is orthogonal, as we have assumed up to now, then we must also have .
• The eigenvalues of a projector are only 1 and 0. For an eigenvalue ,

     

• It exists a basis of such that it is possible to write as , with being the rank of . If we further decompose , with being and being , the existence of the basis shows that really sends points from into and points from into . It also shows that .

Model Order Reduction

Is there any application of projection matrices to applied math? Indeed.

It is often the case (or, at least, the hope) that the solution to a differential problem lies in a low-dimensional subspace of the full solution space. If some is the solution to the Ordinary Differential Equation

then there is hope that there exists some subspace , s.t. in which the solution lives. If that is the case, we may rewrite it as

for some appropriate coefficients , which are the components of over the basis .

Assuming that the base itself is time-invariant, and that in general will be a good but not perfect approximation of the real solution, the original differential problem can be rewritten as:

where is an error.

The post Projection methods in linear algebra numerics appeared first on Quick Math Intuitions.

Every matrix can be seen as a linear map between vector spaces. Stating that a matrix is similar to a diagonal matrix equals to stating that there exists a basis of the source vector space in which the linear transformation can be seen as a simple stretching of the space, as re-scaling the space. In other words, diagonalizing a matrix is the same as finding an orthogonal grid that is transformed in another orthogonal grid. I recommend this article from AMS for good visual representations of the topic.

Diagonalization on non full rank matrices

That’s all right – when we have a matrix from in , if it can be diagonalized, we can find a basis in which the transformation is a re-scaling of the space, fine.

But what does it mean to diagonalize a matrix that has null determinant? The associated transformations have the effect of killing at least one dimension: indeed, a x matrix of rank has the effect of lowering the output dimension by . For example, a 3×3 matrix of rank 2 will have an image of size 2, instead of 3. This happens because two basis vectors are merged in the same vector in the output, so one dimension is bound to collapse.

Let’s consider the sample matrix

which has non full rank because has two equal rows. Indeed, one can check that the two vectors go in the same basis vector. This means that instead of 3. In fact, it is common intuition that when the rank is not full, some dimensions are lost in the transformation. Even if it’s a 3×3 matrix, the output only has 2 dimensions. It’s like at the end of Interstellar when the 4D space in which Cooper is floating gets shut.

However, is also a symmetric matrix, so from the spectral theorem we know that it can be diagonalized. And now to the vital questions: what do we expect? What meaning does it have? Do we expect a basis of three vectors even if the map destroys one dimension?

Pause and ponder.

Diagonalize the matrix and, indeed, you obtain three eigenvalues:

The eigenvalues are thus , and , each giving a different eigenvector.  Taken all together, they form a orthogonal basis of . The fact that is among the eigenvalues is important: it means that all the vectors belonging to the associated eigenspace all go to the same value: zero. This is the mathematical representation of the fact that one dimension collapses.

At first, I naively thought that, since the transformation destroys one dimension, I should expect to find a 2D basis of eigenvectors. But this was because I confused the source of the map with its image! The point is that we can still find a basis of the source space from the perspective of which the transformation is just a re-scaling of the space. However, that doesn’t tell anything about the behavior of the transformation, whether it will preserve all dimensions: it is possible that two vectors of the basis will go to the same vector in the image!

In fact, the fact that the matrix has the first and third rows that are the same means that the basis vectors and both go into . A basis of is simply , and we should not be surprised by the fact that those vectors have three entries. In fact, two vectors (even with three coordinates) only allow to represent a 2D space. In theory, one could express any vector that is combination of the basis above as combination of the usual 2D basis , to confirm that .

The post Diagonalizing a matrix NOT having full rank: what does it mean? appeared first on Quick Math Intuitions.

Let’s consider an example. Start with two simple skew lines:

(Observation: don’t make the mistake of using the same parameter for both lines. Each lines exist on its own, there’s no link between them, so there’s no reason why they should should be described by the same parameter. If this doesn’t seem convincing, get two lines you know to be intersecting, use the same parameter for both and try to find the intersection point.)

The directional vectors are:

So they clearly aren’t parallel. They aren’t incidental as well, because the only possible intersection point is for , but when , is at , which doesn’t belong to . It does indeed make sense to look for the line of shortest distance between the two, confident that we will find a non-zero result.

The idea is to consider the vector linking the two lines in their generic points and then force the perpendicularity with both lines.
We will call the line of shortest distance . In our case, the vector between the generic points is (obtained as difference from the generic points of the two lines in their parametric form):

Imposing perpendicularity gives us:

Solving the two simultaneous linear equations we obtain as solution .

This solution allows us to quickly get three results:

1. The equation of the line of shortest distance between the two skew lines: just replace and in with the values found. In our case, .
2. The intersection point between and : just replace in the parametric equation of . In our case, .
3. The intersection point between and : just replace in the parametric equation of . In our case, .

Do you have a quicker method? Share it in the comments!

The post Quick method to find line of shortest distance for skew lines appeared first on Quick Math Intuitions.