KP's LaboratoryKP's Laboratory is my blog where I publish the results of my the technological experiments and the scope is not limited to math, performance and the Linux Kernel
https://sinkap.github.io/
Tue, 29 Dec 2015 02:52:43 +0000Tue, 29 Dec 2015 02:52:43 +0000Jekyll v2.4.0Gaussian Jordan Elimnation<p>This post is going to explain one of basic building blocks for solving “Linear Algebra Equations”. Consider a set of equations:</p>
<script type="math/tex; mode=display">\begin{align}
a_{0,0}x_1 + a_{0,1}x_2 + \cdots + a_{0,M} = b_0 \\\\
a_{1,0}x_1 + a_{1,1}x_2 + \cdots + a_{1,M} = b_1 \\\\
\vdots \hspace{25mm} \vdots \\\\
a_{N,0}x_1 + a_{N,1}x_2 + \cdots + a_{N,M} = b_N \\\\
\end{align}</script>
<p>This is a system on M unknowns \(x_0, x_1 \cdots x_M \) and N equations. Each variable can be thought of a degree of freedom and each equation can be thought of as a constraint. Think about a three variable situation, like a position of a person in a 3-D coordinate. Without any constraints, he has three degrees of freedom in the x, y and z direction. If we are given three equations describing his position(each equation in x, y and z represents a plane in 3-D), we can pin point his co-ordinates in the 3-D space.</p>
<p><img class="image-quarter" src="/img/2012-10-22-gaussian-jordan-elimination/three-planes.png" alt="three-planes" /></p>
<h4 id="validation">Validation</h4>
<ul>
<li>If \(M > N\), the number of unknowns is greater than the number of equations, the system is said to be undetermined and has infinitely many solutions. The solution space can be restricted by Compressed Sensing.</li>
<li>If \(M < N\), the number of equations are greater than the number of variables, the system is said to be overdetermined. Here the general approach is to find the best fit solution (i.e R.M.S error values are a minimum for all equations)</li>
<li>
<p>If \(M = N\), the system is consistent if the following caveats are satisfied:</p>
<ul>
<li>No row should be a linear combination of the other row, this leads to row degeneracy</li>
<li>If all the equations have a certain variable in the exact same linear combination, the system is afflicted by column degeneracy</li>
</ul>
</li>
<li>Both these equations effective result in the removal of a constraint and thus the system becomes indeterminable.</li>
</ul>
<h4 id="pivoting">Pivoting</h4>
<p>In order to obtain more accurate results and reduce round-off errors, a technique called Pivoting is used. Pivoting is done to convert a matrix to its row echelon form.</p>
<h4 id="what-is-row-echelon-form">What is row echelon form?</h4>
<p>A matrix is said to be in row echelon form if:</p>
<ul>
<li>All non zero rows are above the zero rows.</li>
<li>The first non zero number in a row from the left called the Leading coefficient or Pivot should be strictly to the right of the leading coefficient of row above it.</li>
<li>All entries in a column below the leading coefficient must be zero</li>
</ul>
<p>Here is an example of a matrix in row echelon form:</p>
<script type="math/tex; mode=display">% <![CDATA[
\left[ \begin{array}{ccccc} 1 & a_0 & a_1 & a_2 & a_3 \\ 0 & 0 & 1 & a_4 & a_5 \\ 0 & 0 & 0 & 1 & a_6 \end{array} \right] %]]></script>
<p>Pivoting can be done in two ways:</p>
<h5 id="partial-pivoting">Partial Pivoting</h5>
<p>In this the algorithm selects element the largest absolute value and shuffles the rows in such a way that it lies along the diagonal</p>
<h5 id="complete-pivoting">Complete Pivoting</h5>
<p>The algorithm scans the whole matrix for the largest element and shuffles both columns and rows to place the pivot along a diagonal //(a_{ii} //)</p>
<h4 id="the-algorithm">The Algorithm</h4>
<p>We will be using an example matrix to illustrate this Algorithm (which is given in the text-book <strong>Numerical Recipes in C++</strong>:</p>
<script type="math/tex; mode=display">% <![CDATA[
A = \left[ \begin{array}{ccccc} 1 & 2 & 3 & 4 & 5 \\ 2 & 3 & 4 & 5 & 1 \\ 3 & 4 & 5 & 1 & 2 \\ 4 & 5 & 1 & 2 & 3 \\ 5 & 4 & 3 & 2 & 1\end{array} \right] %]]></script>
<p>The equations we aim at solving are:</p>
<script type="math/tex; mode=display">A \cdot Y = I \\
A \cdot X_1 = B_1 \\
A \cdot X_2 = B_2 \\</script>
<p>The algorithm takes two inputs, matrix A (coefficient matrix) and B (solution vector). The inverse of the matrix is returned in A and the variable vector is returned in B.</p>
<h5 id="step-1-finding-the-pivot-element">Step 1: Finding the Pivot Element</h5>
<p>In the first step the algorithm iterates through the matrix and finds the largest element, in the first iteration the pivot element is the largest element of the last row. In our case it comes out to be five and is in the fist column, so there is no need for a column swap, it only needs to be swapped with t he first row. This swap is maintained in a two book-keeping arrays storing the actual position of pivot, so that the result can be restored.</p>
<p>The next time the algorithm searches for a Pivot element, it excludes \(R_1\) and \(C_1\) from the search.</p>
<p><img class="image-quarter" src="/img/2012-10-22-gaussian-jordan-elimination/pivot11.png" alt="pivot11" /></p>
<p><img class="image-quarter" src="/img/2012-10-22-gaussian-jordan-elimination/pivot21.png" alt="pivot21" /></p>
<h5 id="step-2-normalizing-the-row">Step 2: Normalizing the row</h5>
<p>Before we understand the first step we need to understand why this actually works. Using our transformations we are basically converting the matrix into the identity matrix I. Therefore,</p>
<script type="math/tex; mode=display">if A = I; I \cdot X_1 = B_1^\prime \implies X_1 = B_1^\prime</script>
<p>Where \(B_1^\prime\) is the transformed solution vector</p>
<p>As we are using the equation \(A \cdot Y = I\) to determine the inverse of the matrix we store the result back in A.</p>
<p>This step can be further subdivided into two sub-steps:</p>
<ul>
<li>
<p>Normalize a given row by the Pivot element, So now our matrix equations looks like:</p>
<script type="math/tex; mode=display">% <![CDATA[
A = \left[ \begin{array}{ccccc} \frac{5}{5} & \frac{1}{5} & \frac{2}{5} & \frac{3}{5} & \frac{4}{5} \\ 2 & 3 & 4 & 5 & 1 \\ 3 & 4 & 5 & 1 & 2 \\ 4 & 5 & 1 & 2 & 3 \\ 1 & 2 & 3 & 4 & 5\end{array} \right] \cdot Y = \left[ \begin{array}{ccccc} \frac{1}{5} & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 1\end{array} \right] %]]></script>
<p>The solution vector also gets transformed as:</p>
<script type="math/tex; mode=display">\left[ \begin{array}{c} \frac{b_0}{5} \\ b_1 \\ b_2 \\ b_3 \\ b_4\end{array} \right]</script>
</li>
<li>
<p>The next step is to reduce each element below the Pivot by subtracting the right amount of first row:</p>
<script type="math/tex; mode=display">% <![CDATA[
A = \left[ \begin{array}{ccccc} 1 & 0.2 & 0.4 & 0.6 & 0.8 \\ 0 & 2.6 & 3.2 & 3.8 & -0.6 \\ 0 & 3.4 & 3.8 & -0.8 & -0.4 \\ 0 & 4.2 & -0.6 & -0.4 & 0.2 \\ 0 & 1.8 & 2.6 & 3.4 & 4.2\end{array} \right] \cdot Y = \left[ \begin{array}{ccccc} 0.2 & 0 & 0 & 0 & 0 \\ -0.4 & 1 & 0 & 0 & 0 \\ -0.6 & 0 & 1 & 0 & 0 \\ -0.8 & 0 & 0 & 1 & 0 \\ -0.2 & 0 & 0 & 0 & 1\end{array} \right] %]]></script>
</li>
</ul>
<p>and similar transforms are performed on the solution vector</p>
<p>We, will discuss certain parts of the second iteration as they are slightly different from the first:</p>
<p><img class="image-quarter" src="/img/2012-10-22-gaussian-jordan-elimination/pivot.png" alt="pivot" /></p>
<p>Now while iterating for the second column, the largest element found its at \( R_4,C_4\)</p>
<p>Here, there is no need for swapping as the pivot is found along the diagonal itself.
At the end we have done pivoting for all columns and have reduced our matrix, but we need to accommodate for the shuffling that we have done. Let us say that our book-keeping arrays are:</p>
<p><img class="image-quarter" src="/img/2012-10-22-gaussian-jordan-elimination/arrays.png" alt="arrays" /></p>
<p>Let us take the first case:</p>
<p>As the row and column number was not the same, there is an initial swap that needs to restored back. So, we swap \(C_4\ with\ C_0\). A row operation in the input appears as a column operation in its inverse a (explains the shuffling of columns instead of rows)</p>
Mon, 22 Oct 2012 11:34:20 +0000
https://sinkap.github.io/mathematics/2012/10/22/gaussian-jordan-elimination.html
https://sinkap.github.io/mathematics/2012/10/22/gaussian-jordan-elimination.htmlmathematics