When a connection cannot be established between a client and server using TLS/SSL protocol, an SSLv3 alert handshake failure will occur.

We have helped customers solved SSL related issues in the past as part of our Server Support Services.

In this context we shall look into what triggers this error and how to fix it.

More about SSLv3 alert handshake failure

Handshake possibility in TLS/SSL helps the client and server to establish connection by means of secret keys. This process involves the ability of the client and server to use the same version of the protocol, the cryptographic algorithm and how they authenticate by validating and exchanging digital certificates.

When the handshake failure occurs, the client application will get an HTTP status 503 message signifying "Service Unavailable".

You can see the following error message;

HTTP/1.1 503 Service Unavailable

Or during an TLS/SSL handshake failure, you will see;

Received fatal alert: handshake_failure

This error can result from the following;

i. Server Name Indication (SNI) Enabled Server.
ii. Certificate Incorrect.
iii. Server does not support the Cipher Suite used.
iv. Protocol used not supported.

How to solve SSLv3 alert handshake failure

TLS/SSL handshake failure happens when the protocol used by a client during a connection is not supported by the server.
If this is the case, you can use the tcpdump utility to get to the root of this problem. Run the tcpdump command below;

tcpdump -i any -s 0 host IP address -w File name

Once you get more information, a quick fix would be to perform an upgrade of the server to support TLSv1.2 protocol as this is more secure. If the upgrade process fails then the Message Processor can be forced to use TLSv1.0 protocol for communication with the server.
If you did not specify a target server in the proxy's TargetEndpoint definition, you can assign the Protocol element to TLSv1.0 as shown below;

<TargetEndpoint name="default">

 …

 <HTTPTargetConnection>

   <SSLInfo>

       <Enabled>true</Enabled>

       <Protocols>

           <Protocol>TLSv1.0</Protocol>

       </Protocols>

   </SSLInfo>

   <URL>https://myservice.com</URL>

 </HTTPTargetConnection>

 …

</TargetEndpoint>

A mismatch in the Cipher suite algorithm between the client and the server can also result in a TLS/SSL handshake failure. To fix this, make sure that the cipher suite algorithms used by the client is supported by the server.

In other scenarios, an Incorrect Certificate can cause this error if used in the truststore/keystore. To fix this, simply get a certificate where the CN has a wildcard certificate and the upload to the keystore as shown below;

"subject": "CN=*.domain.com, OU=Domain Control Validated, O=*.domain.com",

Finally, this error can also result when the client is communicating with a Server Name Indication (SNI) Enabled Server in a situation where the client is not SNI enabled. If this is the case, you have to know the hostname and port number of the server you are using to see if it is SNI enabled.
You can use the command below to identify the SNI enabled server;

openssl s_client -connect hostname:port

An easy fix would be to locate the message properties file and add the following attributes;

conf_system_jsse.enableSNIExtension=true

Then give the user permission via the chown command.

Need support in fixing SSL errors? Let us know.