6.6 TLS handshake

Once a session has been initialized and a network connection has been set up, TLS and DTLS protocols perform a handshake. The handshake is the actual key exchange.

Function: int gnutls_handshake (gnutls_session_t session)

session: is a gnutls_session_t type.

This function performs the handshake of the TLS/SSL protocol, and initializes the TLS session parameters.

The non-fatal errors expected by this function are: GNUTLS_E_INTERRUPTED , GNUTLS_E_AGAIN , GNUTLS_E_WARNING_ALERT_RECEIVED . When this function is called for re-handshake under TLS 1.2 or earlier, the non-fatal error code GNUTLS_E_GOT_APPLICATION_DATA may also be returned.

The former two interrupt the handshake procedure due to the transport layer being interrupted, and the latter because of a "warning" alert that was sent by the peer (it is always a good idea to check any received alerts). On these non-fatal errors call this function again, until it returns 0; cf. gnutls_record_get_direction() and gnutls_error_is_fatal() . In DTLS sessions the non-fatal error GNUTLS_E_LARGE_PACKET is also possible, and indicates that the MTU should be adjusted.

When this function is called by a server after a rehandshake request under TLS 1.2 or earlier the GNUTLS_E_GOT_APPLICATION_DATA error code indicates that some data were pending prior to peer initiating the handshake. Under TLS 1.3 this function when called after a successful handshake, is a no-op and always succeeds in server side; in client side this function is equivalent to gnutls_session_key_update() with GNUTLS_KU_PEER flag.

This function handles both full and abbreviated TLS handshakes (resumption). For abbreviated handshakes, in client side, the gnutls_session_set_data() should be called prior to this function to set parameters from a previous session. In server side, resumption is handled by either setting a DB back-end, or setting up keys for session tickets.

Returns: GNUTLS_E_SUCCESS on a successful handshake, otherwise a negative error code.

Function: void gnutls_handshake_set_timeout (gnutls_session_t session, unsigned int ms)

session: is a gnutls_session_t type.

ms: is a timeout value in milliseconds

This function sets the timeout for the TLS handshake process to the provided value. Use an ms value of zero to disable timeout, or GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT for a reasonable default value. For the DTLS protocol, the more detailed gnutls_dtls_set_timeouts() is provided.

This function requires to set a pull timeout callback. See gnutls_transport_set_pull_timeout_function() .

Since: 3.1.0

In GnuTLS 3.5.0 and later it is recommended to use gnutls_session_set_verify_cert for the handshake process to ensure the verification of the peer’s identity. That will verify the peer’s certificate, against the trusted CA store while accounting for stapled OCSP responses during the handshake; any error will be returned as a handshake error.

In older GnuTLS versions it is required to verify the peer’s certificate during the handshake by setting a callback with gnutls_certificate_set_verify_function, and then using gnutls_certificate_verify_peers3 from it. See Certificate authentication for more information.

void gnutls_session_set_verify_cert (gnutls_session_t session, const char * hostname, unsigned flags)

int gnutls_certificate_verify_peers3 (gnutls_session_t session, const char * hostname, unsigned int * status)