Does Server location really matter?

In short, yes it does.
There are many reasons why you should carefully choose the location of your server or host. The main reason being speed.

There’s no doubt that the closer to the server you are geographically, the faster it will serve files and web pages.

The other advantage of choosing a host in the same country as your visitors is search engine rankings.
Google and other search engines take note of the IP address that your site is hosted on and ranks it accordingly.
For example, a website on a UK IP address will rank better on than an American IP address would.