Java Client/SSL.jar interop problems with SSLEAY.

Arjun Khanna (arjun@jump.net)
Fri, 02 Oct 1998 21:57:21 -0500

--=====================_907401441==_
Content-Type: text/plain; charset="us-ascii"

Hi.
I am continuing to do some testing of the SSL.jar against an SSLEAY0.9.0b
based server. My goal is to build a prototype to demonstrate a java client
interop with SSLEAY. Dave Brownell pointed out in an earlier message that
the ssl engine is not meant to be used out side the scope of the browser. I
would like to point out that my intent is to prototype, not build a product
without appropriately licensing the technology from Sun or phaos- if it
works for me. Phaos' evaluation version does not come with DSS support.
Anyways, here are my findings:

I am using a c server which uses SSLEAY to do handle security. I have
turned off client verification on the server to curtail the server from
asking the client for its cert. The server is using a DSA cert, generated
using SSLEAY. The server works just fine against a ssleay client. The
certificate used by the server is in the PEM format which the SSLEAY
library reads in just fine.

The problem arises when I bring up a simple java client (JDK1.2 beta4 +
HotJava ssl.jar ) - file attached against the SSLEAY server- the java
client prints an exception message _before_ the ssl key exchange terminates
and exits (see the key exchange notes below): Here is the msg:

Exception while reading/writing data Server key,
java.security.SignatureException: invalid encoding for signature
javax.net.ssl.SSLException: Server key, java.security.SignatureException:
invalid encoding for signature
at
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:157)
at sun.security.ssl.Handshaker.process_record(Compiled Code)
at sun.security.ssl.SSLSocketImpl.clearPipeline(Compiled Code)
at sun.security.ssl.SSLSocketImpl.write(Compiled Code)
at sun.security.ssl.AppOutputStream.write(Compiled Code)
at DH.<init>(Compiled Code)
at DH.main(DH.java:77)

I have SSL state debugging turned on: When the c/c++ ssleay client is
working against the server: the SSL exchange goes like so (the client's
cert is not being validated, the print out is from the server's perspective):

SSLv3 read client hello A
SSLv3 write server hello A
SSLv3 write certificate A
SSLv3 write key exchange A
SSLv3 write server done A
SSLv3 flush data

SSLv3 read client key exchange A
SSLv3 read finished A
SSLv3 write change Cipher Spec A
SSLv3 write finished A
SSLv3 flush data

After this exchange application data is exchanged and all goes well.

When I run the Java application against the SSLEAY server- I get the
following trace from the SSL state:

SSLv3 read client hello A
SSLv3 write server hello A
SSLv3 write certificate A
SSLv3 write key exchange A
SSLv3 write server done A
SSLv3 flush data

SSL3: alert read fatal handshake failure ssl\s3_pkt.c :791

----
Atleast the trace makes sense somewhat- since the client is printing an
exception that it cannot understand the server key- "invalid encoding for
signature" - its unlikey that the client will respond with the message that
the server is now accepting namely:

SSLv3 read client key exchange A

My question is - why is the client not able to understand the server key?

Thanks in advance for any suggestions/assistance.
-Arjun Khanna
mailto:arjun@jump.net

I'd be happy to mail the secserver file and certs if anyone is interested.

--=====================_907401441==_
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: attachment; filename="DH.java"

import java.io.*;
import java.net.*;
import javax.net.SocketFactory;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;

public class DH {
private static String []
enabledSuites = {
"SSL_DHE_DSS_WITH_DES_CBC_SHA"
};

public DH(String host, int port, int iterations) {
Host = host;
Port = port;
iters = iterations;

SocketFactory factory = SSLSocketFactory.getDefault();

/*
String[] suites = ((SSLSocketFactory)factory).getDefaultCipherSuites();
if(suites != null) {
for (int i = 0; i < suites.length; i++) {
System.out.println("Default Suite : " + suites[i]);
}
}
*/

String[] suites = ((SSLSocketFactory)factory).getSupportedCipherSuites();
if(suites != null) {
for (int i = 0; i < suites.length; i++) {
System.out.println("Supported Suite : " + suites[i]);
}
}

try {
socket = factory.createSocket(Host, Port);
((SSLSocket)socket).setEnabledCipherSuites(enabledSuites);
((SSLSocket)socket).setUseClientMode(true);
((SSLSocket)socket).setEnableSessionCreation(true);
//((SSLSocket)socket).addHandshakeCompletedListener(elistener);
((SSLSocket)socket).startHandshake();
}
catch(Exception e) {
System.out.println("Exception: " + e.getMessage());
e.printStackTrace();

}

try {

int i = 0;
byte[] buf = new byte[2];
buf[0] = 'a' & 0xff;
buf[1] = 'b' & 0xff;
OutputStream s = socket.getOutputStream();
DataInputStream r = new DataInputStream(socket.getInputStream());
while (i++ < iters)
s.write(buf,0,2);

//int x = r.readInt();
}
catch(Exception e){
System.out.println("Exception while reading/writing data " +
e.getMessage());
e.printStackTrace();
}

}

public static void main(String args[]) {

String Host = args[0];
int Port = Integer.parseInt(args[1]);

DH dh = new DH(Host, Port, Integer.parseInt(args[2]));

}

private Socket socket;
private String Host;
private int Port;
int iters;
}

--=====================_907401441==_--