Actually, Phi was not considered a GPU. It was always a compute accelerator. GPU side of that died with Larrabee.
Phi is considered a GPU and it was competing against NV's Tesla.
Where did you get that Xe is derived from Xeon?
We know enough about Xe to say it is not doing x86.
Well that Xe derivative from Xeon is my guess not something legit. If it turns out that the Xe is capable of x86. Im not saying is is capable now but it might be capable. Still it is similar to Phi with the concept in what this Xe GPU will do and what NV and AMD counterparts will go against.
I brought up money because there is a lot of that in compute/AI market. And you said Xe not coming to desktop at first is a sign Intel feels weak. When you are designing a new GPU, that is the obvious market to go for. Desktop GPUs are dirt cheap and require a lot of twitchy games-specific software work.
And money is obvious but feeling weak in one area doesn't mean you can't get money in the other and be good at it. Desktop market (dirt as you said) is still a market and exists and Intel is not competing with NV and AMD here so there must be a reason for that. How I see it and shared it is, Intel can't make a decent GPU for gaming now. It can be because of soft support and that is why Xe is not gaming GPU to get more tracktion etc. Anyway Intel was going to release a desktop GPU (as far as I remember) but it didn't happen.