If you absolutely do not want to use SSL with certificates, you can minimize your own, although this is clearly not so secure. I'm just improvising here, mixing a little asymmetric crypto with a knock port.
First, create a random RSA key pair in the client, in memory, no need to store it anywhere. Then, the client connects to the server using a simple Socket, and when connected, sends the server a public key (encode as you wish, so that you can easily read it on the server). The server then generates a random 128-bit key, launches the ANOTHER ServerSocket on a random port and encrypts the 128-bit key and the new server port number using the client’s public key, and sends the data back to the client. The server must wait a short period of time to receive a connection from the same client on the new port.
The client closes the connection, decrypts the data, and opens a new Socket server on the specified port. Then both the client and server should wrap the InputStream and OutputStream socket on CipherInputStream and CipherOutputStream using AES / CBC / PKCS5Padding (or RC4, if you want) with the specified 128-bit key. Voilá, you have a secure connection between the client and server without authentication.
If you want to handle authentication, you can do it over a secure connection or on the first connection the server can have a pair of RSA keys, they exchange keys, and the server can send a call to the client (the server sends a call to the client using the client’s public key, and the client answers the call using a public server key). But this is unnecessarily complicated, and I think that you still better use standard SSL with key stores ... maybe you can create a KeyStore in memory on the server and send it to the server, as described above, using the first connection (encrypt the keystore with the public key of the server), and then you can configure a second connection to use SSL with this keystore, which is temporary and will be discarded when the client disconnects.
source share