A set S is composed of 20 multi-colored objects

[Mr Greene]

In which case I would assume that the 4th selection would be non-blue as if the colour doesn't matter the answer is 1.

Within the parameters given, the answer cannot be 1. The colour matters by the definition of the question. 

If one were tasked with picking 4 objects from a set of 20 objects, then 1 would be a reasonable answer for the probability. Anyone could pick 4 from an arbitrary group of twenty objects given no further criteria. That is, however, not the task at hand. 

It is true that if one has already picked three blues, then the choice of the fourth is irrelevant at that point since three blues are already chosen. However, that is not the only possible scenario.

One cannot simply select a preferred scenario arbitrarily out of many scenarios and declare that to be the only scenario.

As posed, the colour DOES matter, since it matters whether one reaches into the imaginary bag/box/container/whatever and retrieves either a blue object or a not blue object.

If one selects the scenario that three blue objects have already been chosen, then the fourth object is irrelevant. Three blue objects have been chosen already. The criteria are satisfied. It matters not what colour the fourth one is.

But what if the first one is yellow, or red, or green, or any other colour?

Surely you can see that there are more possibilities, no?

The scenario specifies 3 blue objects are picked. If the fourth can be any colour including blue then no matter what position the 'any-colour' object is picked the chances are 20/20, 19/19, 18,18 or 17/17. in any case the the answer is 1 and multiplying that by the chances for the blue selections would be a pointless exercise.
The extra selection must therefore be 'not-blue' if the scenario has any purpose.
This would insert a (12/X) selection into the matrix with four permutations.
It seems similarly pointless if the object is returned to the pool before next draw.