It depends on the operating system you are using:
1) On windows the safe way of doing this is to download putty from the official web site: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html , start putty and fill the fields with the IP address and port like in this picture:


2) On Linux operating systems you can do this from a terminal, example: ssh user@vps_ip_address -p vps_port