\(x^2+y^2\)がとりうる値の範囲を\(\mathcal{R}\)とおく.
\begin{align*}
&k \in \mathcal{R}\\
\Longleftrightarrow~&\exists x \exists y \begin{cases} x^2+y^2=k \\ 4x^2-8xy+10y^2=1\end{cases}\\
\Longleftrightarrow~&\exists x \exists y \left[\begin{cases} x^2+y^2=k \\ 4x^2-8xy+10y^2=1\end{cases} \land \exists r \exists \theta \begin{cases} x=r\cos\theta \\ y= r\sin \theta \end{cases}\right]\\
\Longleftrightarrow~&\exists x \exists y \exists r \exists \theta \left[\begin{cases} x^2+y^2=k \\ 4x^2-8xy+10y^2=1\end{cases} \land \begin{cases} x=r\cos\theta \\ y= r\sin \theta \end{cases}\right]\\
\Longleftrightarrow~&\exists x \exists y \exists r \exists \theta \left[\begin{cases} r^2=k \\ 4r^2\cos^2\theta-8r^2\cos\theta\sin\theta +10r^2\sin^2\theta=1\end{cases} \land \begin{cases} x=r\cos\theta \\ y= r\sin \theta \end{cases}\right]\\
\Longleftrightarrow~&\exists x \exists y \exists r \exists \theta \left[\begin{cases} r^2=k \\ 4k\cos^2\theta-8k\cos\theta\sin\theta +10k\sin^2\theta=1\end{cases} \land \begin{cases} x=r\cos\theta \\ y= r\sin \theta \end{cases}\right]\\
\Longleftrightarrow~&\exists r \exists \theta \left[\begin{cases} r^2=k \\ 4k\cos^2\theta-8k\cos\theta\sin\theta +10k\sin^2\theta=1\end{cases} \land \exists x \exists y \begin{cases} x=r\cos\theta \\ y= r\sin \theta \end{cases}\right]\\
\Longleftrightarrow~&\exists r \exists \theta \begin{cases} r^2=k \\ 4k\cos^2\theta-8k\cos\theta\sin\theta +10k\sin^2\theta=1\end{cases}\\
\Longleftrightarrow~&\exists r \begin{cases} r^2=k \\ \exists \theta [4k\cos^2\theta-8k\cos\theta\sin\theta +10k\sin^2\theta=1]\end{cases}\\
\Longleftrightarrow~&\exists r \begin{cases} r^2=k \\ \exists \theta [7k-4k\sin 2\theta-3k\cos 2\theta=1]\end{cases}\\
\Longleftrightarrow~&\exists r \begin{cases} r^2=k \\ \exists \theta [3k\cos 2\theta + 4k\sin 2\theta=7k-1]\end{cases}\\
\Longleftrightarrow~&\exists r \begin{cases} r^2=k \\ \exists \theta \left[\left(\begin{array}{c}3k \\ 4k \\ \end{array}\right) \cdot \left(\begin{array}{c}\cos 2\theta \\ \sin 2\theta \\ \end{array}\right)=7k-1 \right]\end{cases}\\
\Longleftrightarrow~&\exists r \begin{cases} r^2=k \\ \exists \alpha \left[5k\cos \alpha =7k-1 \right]\end{cases}\qquad\text{(\(\alpha\)は\(\left(\begin{array}{c}3k \\ 4k \\ \end{array}\right)\)と\(\left(\begin{array}{c}\cos 2\theta \\ \sin 2\theta \\ \end{array}\right)\)のなす角)}\\
\Longleftrightarrow~&\exists r \begin{cases} r^2=k \\ -1 \leq \frac{7k-1}{5k} \leq 1\end{cases}\\
\Longleftrightarrow~&\exists r [r^2=k] \land -1 \leq \frac{7k-1}{5k} \leq 1\\
\Longleftrightarrow~& k \geq 0 \land -1 \leq \frac{7k-1}{5k} \leq 1\\
\Longleftrightarrow~& k > 0 \land -5k \leq 7k-1 \leq 5k\\
\Longleftrightarrow~& k > 0 \land \frac{1}{12} \leq k \leq \frac{1}{2}\\
\Longleftrightarrow~& \frac{1}{12} \leq k \leq \frac{1}{2}\\
\end{align*}
ゆえに,最大値\(\frac{1}{2}\),最小値\(\frac{1}{12}\).
\(\ast\) \(\ast\) \(\ast\)
はじめに\(x=r\cos \theta,y=r\sin \theta\)とおき,そして「積の和(1次結合)」を「内積」と見なして処理してみました(合成でもいいと思いますが).前回の解法と違い,どの行も同値変形なので逆の考察は必要ありません.たぶんこっちの解法のほうが簡単だと思いますがどうでしょう.