Handshake error in IQ Service Test Connect

Identity Security Cloud (ISC) ISC Discussion and Questions
,
1 
Hello everyone, how are you?

I'm having this error in AD, how do I fix it? I've tried a few things but to no avail.

Thank you.

09/19/2025 14:02:40 : RpcHandler [ Thread-6 ] ERROR : “An Exception occurred while accepting new client requestSystem.IO.IOException: Falha do handshake devido a um formato de pacote inesperado.
em sailpoint.rpcserver.RpcHandler.AuthenticateServer(X509Certificate2 serverCertificate, SslProtocols sslprotocol, Boolean initialCall)
em sailpoint.rpcserver.RpcHandler..ctor(Hashtable services, Hashtable registry, TcpClient client, String port, Boolean useTLS, String subject, String tlsVersion, String registeredClients, String serialNumber)”

2

Hi @henriqueoliveiraIAM ,

  1. Kindly check if RPC services are up and running on both client and servers.

  2. Check if the authenticated user has appropriate access to perform the required opeartions.

  3. Validate if the certs are correctly installed in case of TLS(636 port).

3

Hi @henriqueoliveiraIAM ,

This error usually indicates a TLS/SSL handshake failure between SailPoint and your AD or IQService server caused by a protocol or certificate mismatch. To fix it, please check the following:

  • Ensure TLS versions and cipher suites configured on both ends match exactly.

  • Import all required CA certificates into SailPoint’s truststore to validate the server certificate.

  • If there’s a load balancer in the path, confirm it correctly passes through TLS traffic.

  • Verify the correct ports are used (typically port 443 for HTTPS).

  • Review any NTLM or authentication settings that may affect TLS negotiation.

  • Check firewall rules aren’t blocking or interrupting the connection.

  • After changes, restart IQService and relevant connectors.

Most RpcHandler handshake issues arise from these causes, and addressing them usually resolves the problem.

For detailed troubleshooting, you can refer to SailPoint’s official IQService TLS docs:
https://documentation.sailpoint.com/connectors/iqservice/help/integrating_iqservice_admin/troubleshooting.html

4

This is a TLS/SSL handshake failure between the IQService and SailPoint ISC tenant. It typically points to a mismatch in TLS versions, certificates or encryption.

Discussions regarding the same problem:

Troubleshooting steps:

  1. Check TLS version compatibility:
    Ensure both IQService and the connecting client (e.g., ISC tenant) are configured to use the same TLS version
  2. Verify certificate validity:
    Confirm the server certificate used by IQService is valid, trusted, and not expired. If using a self-signed certificate, make sure it’s trusted by the client.
  3. Review IQService configuration:
    Double-check the useTLS, tlsVersion, subject, and serialNumber settings in the IQService configuration file.
  4. Confirm client configuration:
    Make sure the ISC tenant or other connecting application is set to use TLS and references the correct certificate.
  5. Check for network interference:
    Ensure there are no firewalls, proxies, or network devices modifying or blocking the handshake packets.
  6. Test direct connectivity:
    Try connecting directly (bypassing any intermediaries) to rule out network issues.
  7. Enable debug logging in IQService:

    • Open the IQService configuration file and set the logging level to DEBUG or TRACE.

    • Restart the IQService after making this change.

    • Review the detailed logs for handshake errors, certificate issues, or protocol mismatches.

  8. Restart services:
    After making configuration changes, restart IQService and any related services to apply updates.
5 
even to connect with port 389, do I need to use the TLS certificate?, I'm trying to do it in my lab
6

@henriqueoliveiraIAM pls watch this video - it may help you

7

TLS handshake error against on-prem AD endpoint used by IQService (verify return code 21)

Environment

Windows Server 2016 running IIS (Default Web Site)

Cert bound to https://AD.corp.sailpointlab.com:5050 (same FQDN in CN/SAN)

Client test from Linux/VA using openssl (TLS 1.2)

Symptom

When I test the HTTPS endpoint, the handshake succeeds but certificate validation fails:

openssl s_client -connect AD.corp.sailpointlab.com:5050

Key output:

depth=0 ``CN=AD.corp.sailpointlab.com
verify error:num=20: unable to get local issuer certificate
verify error:num=21: unable to verify the first certificate
Verify return code: 21 (unable to verify the first certificate)
Certificate chain shown by server:
0 s: ``CN=AD.corp.sailpointlab.com
i: DC=com, DC=sailpointlab, DC=corp, CN=corp-AD-CA

*I’m currently testing this in a lab environment, and I haven’t been able to resolve this case yet. Any guidance from the community would be very helpful.

Help me*

1 Like