You should refrain from changing the return type based on the values of the input parameters (as opposed to, say, a static type known at compile time). This makes the caller do part of your method's job, i.e figuring out what has been returned.
There are two solutions to this:
- Always return an object of the same type - Your method would return either an NxN matrix, or a 1x1 matrix, depending on whether the column or the row comes first, or
- Define separate methods with different types - Instead of using the operator, make
MultiplyRowCol
andMultiplyColRow
methods, returning objects of different kinds.
Technically speaking, the first solution is better, because a multiplication of a row by column produces a matrix with a single element, not a scalar.