A digital signature is the process of calculating a digest (function H) of data (C) and encrypting it using an asymmetric encryption algorithm (function E) to create the text cypher (S):
S = E(H(C))
Signature verification accepts the signature, decrypts the given signature (function D), which leads to H (C) only if the public key used in decryption is paired with the private key used in encryption and calculates the data digest to verify that two digests match :
H(C) == D(E(H(C)))
From this it is clear that the bytes specified by the hash function (C) must be exactly the same for the signature to be checked.
In your case, this is not so, because when you calculate the digest using openssl dgst , the output (H (C) on the right) literally looks something like this:
SHA1(someHTMLDoc.html)= 22596363b3de40b06f981fb85d82312e8c0ed511
And this is the input for RSA encryption.
And when you check the signature, the digest output (H (C) on the left) is raw bytes, for example, in hexadecimal format:
22596363b3de40b06f981fb85d82312e8c0ed511
So you will end up encrypting the bytes to create (H (C) on the right):
0000000: 5348 4131 2873 6f6d 6548 746d 6c44 6f63 SHA1(someHtmlDoc 0000010: 2e68 746d 6c29 3d20 3232 3539 3633 3633 .html)= 22596363 0000020: 6233 6465 3430 6230 3666 3938 3166 6238 b3de40b06f981fb8 0000030: 3564 3832 3331 3265 3863 3065 6435 3131 5d82312e8c0ed511 0000040: 0a .
and comparison with bytes (H (C) on the left):
0000000: 2259 6363 b3de 40b0 6f98 1fb8 5d82 312e " Ycc..@.o... ].1. 0000010: 8c0e d511 ....
You also need to use -sign with openssl dgst to have the correct output format (see The difference between openSSL rsautl and dgst ).
So, on the OpenSSL side, do:
openssl dgst -sha1 -sign privateKey.pem someHTMLDoc.html > signature.bin
On the Java side, do:
import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.InputStream; import java.io.InputStreamReader; import java.security.KeyFactory; import java.security.Signature; import java.security.interfaces.RSAPublicKey; import java.security.spec.X509EncodedKeySpec; import org.spongycastle.util.io.pem.PemObject; import org.spongycastle.util.io.pem.PemReader; public class VerifySignature { public static void main(final String[] args) throws Exception { try (PemReader reader = publicKeyReader(); InputStream data = data(); InputStream signatureData = signature()) { final PemObject publicKeyPem = reader.readPemObject(); final byte[] publicKeyBytes = publicKeyPem.getContent(); final KeyFactory keyFactory = KeyFactory.getInstance("RSA"); final X509EncodedKeySpec publicKeySpec = new X509EncodedKeySpec(publicKeyBytes); final RSAPublicKey publicKey = (RSAPublicKey) keyFactory.generatePublic(publicKeySpec); final Signature signature = Signature.getInstance("SHA1withRSA"); signature.initVerify(publicKey); final byte[] buffy = new byte[16 * 1024]; int read = -1; while ((read = data.read(buffy)) != -1) { signature.update(buffy, 0, read); } final byte[] signatureBytes = new byte[publicKey.getModulus().bitLength() / 8]; signatureData.read(signatureBytes); System.out.println(signature.verify(signatureBytes)); } } private static InputStream data() throws FileNotFoundException { return new FileInputStream("someHTMLDoc.html"); } private static PemReader publicKeyReader() throws FileNotFoundException { return new PemReader(new InputStreamReader(new FileInputStream("publicKey.pem"))); } private static InputStream signature() throws FileNotFoundException { return new FileInputStream("signature.bin"); } }
I used Spongy Castle to PEM-decode the public key to make things more readable and easy to use.