Fixing SSL Configuration Problems in Ubuntu 24.04.1's SOLR 9.6.1 and Zookeeper 3.8.1

Challenges in Enabling SSL for SOLR with Zookeeper Integration

Enabling SSL in a SOLR-Zookeeper setup can be tricky, especially when working with Ubuntu 24.04.1 servers. This configuration process ensures secure communication between nodes, but even a minor misconfiguration can prevent services like SOLR Admin UI from functioning properly. If you've recently tried enabling SSL and encountered issues, you are not alone.

In this article, we will walk through a common problem faced during SSL activation in SOLR 9.6.1 when integrated with Zookeeper 3.8.1 on a local Ubuntu server. The setup in question involves running SOLR and Zookeeper on the same server with a single shard, multiple replicas, and basic authentication. The focus will be on resolving the errors that occur after updating SSL settings.

SSL misconfigurations often result in errors like "Admin UI not launching" or "Broken pipe" messages in log files, which can be challenging to troubleshoot. These errors typically arise from certificate issues or SSL connection failures within the SOLR or Zookeeper nodes, leading to broken communication between services.

In the following sections, we will dive deeper into the log files, analyze the potential causes of these SSL-related errors, and offer step-by-step solutions to ensure a smooth SSL configuration for your SOLR and Zookeeper setup.

Analyzing SSL Configuration and Scripting for SOLR and Zookeeper

The first script automates the process of restarting SOLR and Zookeeper while ensuring SSL configurations are properly applied. It uses Bash scripting to loop through the Zookeeper instances and restart them with updated SSL settings. The importance of this script lies in managing multiple Zookeeper nodes, as SSL configurations must be applied uniformly across the entire cluster. The use of `zkServer.sh restart` ensures that each Zookeeper node is restarted correctly with its respective configuration file, making the script efficient for cluster management in a multi-node setup.

The script also addresses the restart of the SOLR instance using `solr restart`. SOLR relies on Jetty for handling HTTPS requests, and the script ensures that SSL-related settings such as keystore and truststore paths are correctly reloaded. This prevents potential SSL handshake failures when accessing SOLR Admin UI, which can arise from outdated or misconfigured SSL certificates. By automating these tasks, the script minimizes manual errors, especially when managing SSL certificates across multiple services on the same server.

The second script is used to create and manage Java KeyStores for SSL in both SOLR and Zookeeper. Java’s Keytool utility is employed to generate key pairs and import certificates into the keystore. The command `keytool -genkeypair` generates the necessary SSL certificates, while `keytool -import` is used to add trusted root and intermediate certificates. These certificates ensure that SSL communication between nodes is trusted and secure. This script is crucial for correctly setting up and managing SSL certificates, which play a central role in enabling secure communication between the services.

Finally, the Python script provided acts as a log monitoring tool specifically designed to detect SSL handshake errors. By continuously reading SSL logs in real time, this script can identify SSL-related issues such as `SSL handshake failed`. This level of logging is essential for diagnosing problems in complex environments where services like Zookeeper and SOLR communicate over encrypted channels. Real-time monitoring helps in quickly identifying the root cause of SSL failures, which might stem from certificate mismatches, incorrect configuration, or expired certificates. This troubleshooting tool is particularly valuable in environments with multiple nodes and SSL complexities.

#!/bin/bash
# Script to automate SOLR and Zookeeper restart with SSL configuration
# Paths to configuration files
ZOOKEEPER_DIR="/opt/zookeeper"
SOLR_DIR="/opt/solr"
SSL_KEYSTORE="/opt/solr-9.6.1/server/etc/solr-ssl.jks"
ZOOKEEPER_CONFIG="$ZOOKEEPER_DIR/conf/zoo.cfg"
SOLR_CONFIG="$SOLR_DIR/server/etc/jetty-ssl.xml"

# Restart Zookeeper with SSL configuration
echo "Restarting Zookeeper..."
for i in {1..3}; do
    /bin/bash $ZOOKEEPER_DIR/bin/zkServer.sh restart $ZOOKEEPER_DIR/data/z$i/zoo.cfg
done

# Restart SOLR with SSL configuration
echo "Restarting SOLR..."
/bin/bash $SOLR_DIR/bin/solr restart -c -p 8983 -z localhost:2181,localhost:2182,localhost:2183 -m 5g -force

#!/bin/bash
# Generate a keystore with a self-signed certificate
keytool -genkeypair -alias solr -keyalg RSA -keystore /opt/solr-9.6.1/server/etc/solr-ssl.jks

# Import intermediate and root certificates
keytool -import -trustcacerts -alias root -file /path/to/rootCA.pem -keystore /opt/solr-9.6.1/server/etc/solr-ssl.jks
keytool -import -trustcacerts -alias intermediate -file /path/to/intermediateCA.pem -keystore /opt/solr-9.6.1/server/etc/solr-ssl.jks

# Configure Zookeeper SSL settings
echo "ssl.client.enable=true" >> $ZOOKEEPER_DIR/conf/zoo.cfg
echo "ssl.quorum.keyStore.location=/opt/solr-9.6.1/server/etc/solr-ssl.jks" >> $ZOOKEEPER_DIR/conf/zoo.cfg
echo "ssl.quorum.trustStore.location=/opt/solr-9.6.1/server/etc/solr-ssl.jks" >> $ZOOKEEPER_DIR/conf/zoo.cfg

import subprocess
import time

def monitor_ssl_logs(log_file):
    with open(log_file, 'r') as f:
        f.seek(0, 2)  # Move to the end of file
        while True:
            line = f.readline()
            if not line:
                time.sleep(0.1)
                continue
            if "SSL handshake failed" in line:
                print(f"Error: {line.strip()}")

# Start monitoring Zookeeper SSL logs
monitor_ssl_logs("/opt/zookeeper/logs/zookeeper.log")

SSL Handshake and Configuration Complexities in SOLR and Zookeeper

One critical aspect to address when enabling SSL in SOLR and Zookeeper is how the SSL handshake process works. The handshake involves the exchange of certificates between client and server, verifying trust before encrypted data transmission begins. Issues often arise if the certificates are not correctly set in both SOLR and Zookeeper configurations. For instance, mismatched certificate chains or keystore passwords can prevent the system from successfully initiating an SSL connection. SOLR relies on Jetty for managing SSL communication, making it important to ensure that the Jetty configuration is in sync with your keystore settings.

Another common challenge is setting up SSL across multiple nodes, especially in a Zookeeper quorum. With multiple Zookeeper nodes, SSL configuration has to be consistent across all servers to enable secure client-to-server and server-to-server communication. Each node must have the same keystore and truststore setup, as well as identical SSL protocols such as TLSv1.2. These configurations are found in the `zoo.cfg` file. Any discrepancy between the nodes can lead to issues like the "broken pipe" or "socket is closed" errors, as witnessed in the problem scenario.

It's also essential to consider how Zookeeper handles the quorum communications with SSL enabled. By setting `ssl.quorum.enabledProtocols`, you ensure that the secure communication between Zookeeper nodes occurs over a trusted protocol like TLS. Additionally, keeping `ssl.quorum.hostnameVerification=false` might be necessary in cases where Zookeeper nodes are referred to by IP rather than hostnames, as hostname mismatches can interrupt the SSL handshake. Fine-tuning these settings can significantly improve secure communication across your distributed setup.