Im pretty sure the answear is yes

. Tho as to why you would want to do that, i dont know. There is little memory savings if thats what you think of.
XFCE uses very little memory imho, and there's still many ways to save resources when using any gui. I'd look up a gentoo forum and ask there.
I remember seeing some screenies where there was only like IRC program opened and a bash/shell/cli if u will.