However, I couldn't have shot a sharp picture anyway as it seems like the focus is not reaching infinity.
I've found with my RF glass that focusing out of infinity is actually not the best for things like stars. In fact, it's almost like focusing all the way to infinity is like overshooting the right amount of focus. I usually set my camera up for manual focus when doing stars and I'll use the LCD to zoom in to one particularly pronounced star to focus off of and more often than not, focus is close to infinity, but not quite there.
This is an example from last winter, but this is shooting straight up on a clear night at 8s, ƒ4, ISO 1600 scaled down from the max width of 6240 to 5000px. This was with my 24-105mm ƒ4 L series len.
That's unprocessed. Now here is where it gets interesting. Using my (arguably,) inferior nifty 50 (50mm ƒ1.8,) I can get something like this at 4s, ƒ2, ISO 800:
...and with a little bit of post processing of that same image above, you get something like this:
However, if you look at the edges of the frame with the 50mm ƒ1.8, you can clearly see comatic distortion. It looks like this (from the top left corner):
Lower quality lenses have this kind of distortion around the edges wide open, but should be pretty sharp near the center of the frame. If you look at the image using the 24-105 ƒ4, you don't have nearly the same level of comatic distortion around the edges. If the center of the frame is blurry, then it's more likely that the subject is out of focus and since stars are so small, even a slight misfocus will make them look blurry. So it's worth taking the time to digitally zoom into a reference star and very slowly adjust focus until it's perfect on the LCD.
If you zoom in and you see something like this, it means your shutter speed is too slow and the stars are moving in the night sky during the shot. (This was at 105mm @ 10s)
I don't have an example of misfocus handy, but if they don't look elongated near the center, then it's probably misfocus.