In the formula, a and b are the legs and c is the hypotenuse of a right triangle. We are given a triangle with c=6in. and 5-in. leg. Let's call this leg a.
We found that the length of the missing leg is sqrt(11)inches. Since a negative side length does not make sense, we only need to consider positive solutions.