You could encrypt long messages using RSA just like block ciphers do. That is, messages in blocks are encrypted and the blocks are associated with the corresponding chain mode. However, this is not an ordinary way to do this, and you will not find support for it (RSA chain) in the libraries you use.
Since RSA is fairly slow, a common way to encrypt large messages is to use hybrid encryption. Hybrid encryption uses a fast, symmetric encryption algorithm (such as AES ) to encrypt data with a random key. Then the random key is encrypted using RSA and sends the symmetric key along with the encrypted data.
EDIT:
As for your implementation, you have insize = 1300 and keylength = 1024 , which gives maxlen = 117 . To encrypt a complete message, you need 12 ciphers, each of which creates 128 bytes, providing an encrypted size of 1536 bytes. In your code, you only allocate 1416 byte buffers. Also, you don't seem to allow 128 bytes to be output, as you only increment with 117 in:
RSA_public_encrypt(maxlen, buff1+i, buff2+i, keypair, RSA_PKCS1_PADDING)
and
i += maxlen;
source share