Installation: Red Hat 8

Introduction

These installation instructions are a guide to installing an OCS reporting server built on Apache Nifi, PostgreSQL and Apache Superset. It is assumed by these instructions that the target platform is Red Hat 8.

It is strongly recommended that the following instructions are used to deploy the reporting services on to one or more dedicated hosts. It is particularly crucial that the reporting services are not installed on to an node that is running real time call control or OCS servers.

Pre-Requisites

These instructions rely on installing a variety of tools into a Linux-based server. The following pre-requisites are assumed for this installation guide:

1. The server OS is Red Hat 8 (any version).
2. For installation of Apache Superset, the server OS has full access to the internet. Apache Nifi and PostgreSQL can be installed with only dnf (or yum access to official Red Hat repositories - or satellite repositories).

Install and Configure Apache

For proxying through standard HTTP ports, install a HTTP proxy. In these instructions Apache HTTPD is used:

dnf install httpd

Once installed, tweak SELinux. As Red Hat 8 uses SELinux for security, some restrictions must be lifted.

Allow proxying via http to work. As root:

/usr/sbin/setsebool -P httpd_can_network_connect 1

Apache Nifi

Installation

To install Apache Nifi, first install Java:

dnf install java-latest-openjdk

Download the latest Apache Nifi (v1.13 or later) from the Apache Nifi downloads page. Download the binary file:

cd /var/tmp/
VERSION=1.13.2
wget https://apache.inspire.net.nz/nifi/${VERSION}/nifi-${VERSION}-bin.tar.gz

The Nifi toolkit is not required. Next, uncompress and install the package:

cd /opt
tar zvfx /var/tmp/nifi-${VERSION}-bin.tar.gz

Then, install the new version of Nifi, assuming /opt/nifi links to a previous install of Nifi:

rm /opt/nifi
ln -s nifi-${VERSION} nifi

Next, create the Nifi data directory, if it doesn’t exist:

mkdir -p /opt/nifi-data

Configuration

boostrap.conf

The bootstrap configuration file determines the basic variables for running Nifi. Of primary importance is the memory usage for Nifi. Real time processing of OCS EDRs requires sufficient memory for managing a relatively significant stream of data, even on smaller installations:

Edit bootstrap.conf:

vi /opt/nifi/conf/bootstrap.conf

Update at least the following variables:

zookeeper.properties

Nifi uses Apache Zookeeper to handle Nifi node clustering. Nifi clustering is out of scope of this installation guide, however the Zookeeper configuration must still be correct.

Edit the zookeeper configuration:

vi /opt/nifi/conf/zookeeper.properties

Update at least the following variables:

nifi.properties

Apache Nifi uses a properties file to determine working directories and other system-specific configuration:

Edit the Nifi configuration:

vi /opt/nifi/conf/nifi.properties

Update at least the following variables:

Consider also changing nifi.web.http.host to the local IP (not loopback) for access, however only perform this change if a HTTPS relay is not being used, and the host is within a secure environment (as Apache Nifi does not provide secure access).

stateless.properties

The final Nifi configuration file is “stateless” configuration. Edit the file:

vi /opt/nifi-data/conf/stateless.properties

Startup Configuration

To run Nifi automatically on system startup, configure and enable Superset using systemd. As root, place the following file contents in /etc/systemd/system/nifi.service:

[Unit]
Description=Apache NiFi
After=network.target

[Service]
Type=forking
User=root
Group=root
ExecStart=/opt/nifi/bin/nifi.sh start
ExecStop=/opt/nifi/bin/nifi.sh stop
ExecRestart=/opt/nifi/bin/nifi.sh restart 

[Install]
WantedBy=multi-user.target

Then to ensure the new configuration is loaded, run:

systemctl daemon-reload

Web Proxy

It is trivial to proxy to Apache Nifi through Apache HTTP (or any other web proxy). The following apache configuration can be placed in a file such as /etc/httpd/conf.d/nifi.conf:

#
# N-Squared example Nifi proxy configuration
#
<VirtualHost *:80>
    #ServerName nifi.nsquared.co.nz

    RedirectMatch ^/$ /nifi

    # Everything is proxied
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "http"
    # RequestHeader add X-ProxyHost "nifi.nsquared.co.nz"
    RequestHeader set X-Forwarded-Port "80"
    ProxyPass / http://127.0.0.1:8080/
    ProxyPassReverse / http://127.0.0.1:8080/

</VirtualHost>

Then restart apache:

systemctl restart httpd

BI Database Installation

The N-Squared OCS reporting database is deployed on PostgreSQL and utilises DbMaintain for schema maintenance. DbMaintain is a simple tool for managing databases as they change over time.

Install PostgreSQL

The reporting database requires PostgreSQL to be installed:

dnf install dnf install postgresql-server

Create the postgres instance and enable:

postgresql-setup --initdb
systemctl enable postgresql
systemctl start postgresql

Change the auth approach for postgres to allow username/password logins:

vi /var/lib/pgsql/data/pg_hba.conf

and change:

host    all             all             127.0.0.1/32            ident
host    all             all             ::1/128                 ident

to:

host    all             all             127.0.0.1/32            md5
host    all             all             ::1/128                 md5

Once installed, the reporting database can be created. For the reporting database, the OCS uses the following schemas which will be created in the n2reporting database by the following instructions:

• n2ocs - The reporting schema for the OCS.

The content of this schema is then able to be created by the SQL in the /opt/nsquared/n2ocs/db/reportingdb/sql directory.

Database Creation

Edit the configuration file /opt/nsquared/ocs/db/reportingdb/dbmaintain.properties to configure the connection to the reporting database (i.e. usename, password, URL).

However, before running dbmaintain the first time, the database itself needs creation:

1. Change user to the postgres user:

   sudo su - postgres
   

2. Create a report writer, and report reader user for the database. Note that report readers may be configured based on organisation procedures other than shown here - however following instructions to configure Nifi and Superset connectivity will require changing. You will need to type in the password:

   createuser -P n2ocs_report_writer
   createuser -P n2ocs_report_reader
   

3. Create the database (if it doesn’t already exist):

   createdb -O n2ocs_report_writer n2reporting
   

4. Create the database schema, owned by the report writer user.

   psql n2reporting
   CREATE SCHEMA n2ocs AUTHORIZATION n2ocs_report_writer;
   

5. Confirm database login works, and set the schema search path.

   psql -h localhost -U n2ocs_report_writer n2reporting
   ALTER ROLE n2ocs_report_writer SET search_path TO n2ocs,public;
   

   psql -h localhost -U n2ocs_report_reader n2reporting
   ALTER ROLE n2ocs_report_reader SET search_path TO n2ocs,public;
   

6. As the database superuser, grant login:

   psql n2reporting
   ALTER ROLE n2ocs_report_writer LOGIN;
   ALTER ROLE n2ocs_report_reader LOGIN;
   

Database Server Creation and Updates

Once the reporting database is created database updates are all performed through DbMaintain. DbMaintain will automatically apply the necessary changes. To run:

2. Edit the reportingdb/dbmaintain.properties file and set database password appropriately.

3. Run:

   cd /opt/nsquared/ocs/db/
   export DBMAINTAIN_JDBC_DRIVER=/usr/share/dbmaintain/postgresql-42.3.1.jar
   /usr/share/dbmaintain/dbmaintain.sh updateDatabase -config reportingdb/dbmaintain.properties
   

Apache Superset

User Creation

To ensure security, create a superset user. As root:

groupadd superset
useradd -d /opt/superset -g superset -m -s `which bash`

then set the superset user password:

passwd superset

Python venv Creation

Ensure that python 3.6 or later is installed:

dnf install python3-rpm python3-pip

Ensure that Python’s pip is as up-to-date as possible. Use the RPM provided PIP to bootstrap a newer version. Do this as the superset user:

su - superset
python3 -m pip install --user --upgrade pip

and then install Python’s virtual environment solution:

python3 -m pip install --user virtualenv

and finally create the superset virtual environment as the superset user:

cd /opt/superset
python3 -m venv env

Install Superset

Install Superset into the created virtual environment. This requires upgrading pip again, and then finally installing Apache Superset. As the superset user:

. env/bin/activate
pip install --upgrade pip
pip install apache-superset
pip install gunicorn # for web server execution

Configure Superset

Most Superset configuration is performed in the GUI. Before the GUI can be accessed, run the following commands. These commands depend on the Superset version installed, so it is recommended that the official Superset installation instructions are checked as well.

As the superset user:

superset db upgrade
export FLASK_APP=superset
superset fab create-admin

The last command will force you to create the first administration user. Set an appropriate administration username and password, then run:

superset init

Manual Startup

For testing and debuggin purposes, Superset can be started from the command line using the command:

superset run --host=0.0.0.0 -p 8088 --with-threads --reload --debugger

This runs Apache Superset using the built-in web server in the foreground on port 8088, accessible across all network interfaces available.

Startup Configuration

To run Superset automatically, configure and enable Superset using systemd. As root, place the following file contents in /etc/systemd/system/superset.service:

[Unit]
Description = Apache Superset Webserver Daemon
After = network.target

[Service]
PIDFile = /opt/superset/superset-webserver.PIDFile
User = superset
Group = superset
Environment=SUPERSET_HOME=/opt/superset
Environment=PYTHONPATH=/opt/superset
WorkingDirectory = /opt/superset
ExecStart =/opt/superset/venv/bin/python3.6 /opt/superset/venv/bin/gunicorn --workers 8 --worker-class gevent  --bind 0.0.0.0:8888 --pid /opt/superset/superset-webserver.PIDFile superset:app
ExecStop = /bin/kill -s TERM $MAINPID


[Install]
WantedBy=multi-user.target

Web Proxy

It is trivial to proxy to Apache Superset through Apache HTTP (or any other web proxy). The following apache configuration can be placed in a file such as /etc/httpd/conf.d/superset.conf:

#
# N-Squared example superset proxy configuration
#
<VirtualHost *:80>
    #ServerName superset.nsquared.co.nz

    RedirectMatch ^/$ /superset/welcome

    # Everything is proxied
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "http"
    RequestHeader set X-Forwarded-Port "80"
    ProxyPass / http://127.0.0.1:8088/
    ProxyPassReverse / http://127.0.0.1:8088/

</VirtualHost>

Then restart apache:

systemctl restart httpd