prove that determinant of (X^2+Y^2)>=0 if X*Y=Y*X and X is not equal to YX and Y are square matrices.

We'll start from the given constraint X*Y=Y*X. We know
that the product of two matrices is not commutative.

Since
X is not equal to Y, then, X*Y=Y*X if and only if Y = X^-1 (Y is the inverse of the
matrix X).

We know that X*X^-1 = I, where I is the identity
matrix.

Since the square matrix X has the inverse X^-1,
then the determinant of the matrix X is different from zero
value.

det X>0 or det X
<0

det (X^2 + Y^2) = det X^2 + det
Y^2

If det X<0, then det X^2 >
0

Since Y = X^-1, then det Y = det X^-1 => det
Y^2>0

Therefore, the given inequality
det (X^2 + Y^2) >= 0.