- $\sin^2\alpha$
Solution:
$\cos^{-1}\text{x}+\cos^{-1}\text{y}=\cos^{-1}\Big(\text{xy}-\sqrt{1-\text{x}^2}\sqrt{1-\text{y}^2}\Big)$
Consider, $\cos^{-1}\frac{\text{x}}{\text{a}}+\cos^{-1}\frac{\text{y}}{\text{b}}=\alpha$
$\Rightarrow\cos^{-1}\bigg(\frac{\text{x}}{\text{a}}\times\frac{\text{y}}{\text{b}}-\sqrt{1-\frac{\text{x}^2}{\text{a}^2}}\sqrt{1-\frac{\text{y}^2}{\text{a}^2}}\bigg)$
$\Rightarrow\frac{\text{x}}{\text{a}}\times\frac{\text{y}}{\text{b}}-\sqrt{1-\frac{\text{x}^2}{\text{a}^2}}\sqrt{1-\frac{\text{y}^2}{\text{a}^2}}=\cos\alpha$
$\Rightarrow\frac{\text{x}}{\text{a}}\times\frac{\text{y}}{\text{b}}-\cos\alpha=\sqrt{1-\frac{\text{x}^2}{\text{a}^2}}\sqrt{1-\frac{\text{y}^2}{\text{a}^2}}$
Squaring on both sides,
$\Rightarrow\frac{\text{x}^2\text{y}^2}{\text{a}^2\text{b}^2}+\cos^2\alpha-\frac{2\text{xy}}{\text{ab}}\cos\alpha=\Big(1-\frac{\text{x}^2}{\text{a}^2}\Big)\Big(1-\frac{\text{y}^2}{\text{a}^2}\Big)$
$\Rightarrow\frac{\text{x}^2\text{y}^2}{\text{a}^2\text{b}^2}+\cos^2\alpha-\frac{2\text{xy}}{\text{ab}}\cos\alpha=1-\frac{\text{x}^2}{\text{a}^2}-\frac{\text{y}^2}{\text{a}^2}+\frac{\text{x}^2\text{y}^2}{\text{a}^2\text{b}^2}$
$\Rightarrow\frac{\text{x}^2}{\text{a}^2}+\frac{\text{y}^2}{\text{a}^2}-\frac{2\text{xy}}{\text{ab}}\cos\alpha=1-\cos^2\alpha$
$\Rightarrow\frac{\text{x}^2}{\text{a}^2}+\frac{\text{y}^2}{\text{a}^2}-\frac{2\text{xy}}{\text{ab}}\cos\alpha=\sin^2\alpha$