From Ian Winter:
Yes, the metal collar and bush should slide off of the top of the
gear-lever. If it's a bit tight then I'd have thought a few taps
around the circumference of the collar with a small punch should drift
it off. I guess you could try a small amount of heat on the collar also
but be careful to protect the nylon bush if you plan to reuse it else it
may melt. Interestingly, your bush must be someone's replacement item as
these are from the Hillman Imp and originally black in colour.
You might wish to replace the bush and collar with the harder material
bush that Malcolm Anderson (Imp parts) supplies, as these should last
longer. see
http://www.malcolmanderson.co.uk/sparesct.html