Apache reverse proxy

could do with filling out more detail

A reverse proxy allows you to front multiple websites from a single public IP address, act as a load balancer and potentially defuse otherwise dangerous cyber attacks. There are different solutions, the instructions here are for an Apache server-based solution. Basic requirements are Apache, mod_ssl and mod_proxy installed and enabled.

Installation

Install apache2

apt-get install apache2

File /etc/apache2/mods-available/proxy.conf should look like this:

<IfModule mod_proxy.c>

        # If you want to use apache2 as a forward proxy, uncomment the
        # 'ProxyRequests On' line and the <Proxy *> block below.
        # WARNING: Be careful to restrict access inside the <Proxy *> block.
        # Open proxy servers are dangerous both to your network and to the
        # Internet at large.
        #
        # If you only want to use apache2 as a reverse proxy/gateway in
        # front of some web application server, you DON'T need
        # 'ProxyRequests On'.

        ProxyRequests Off

        <Proxy *>
                AddDefaultCharset off
                Order deny,allow
                Deny from all
        </Proxy>

        # Enable/disable the handling of HTTP/1.1 "Via:" headers.
        # ("Full" adds the server version; "Block" removes all outgoing Via: headers)
        # Set to one of: Off | On | Full | Block
        #ProxyVia Off

        ProxyVia On
        ProxyPreserveHost On
        ProxyRequests Off
        ProxyTimeout 600

</IfModule>

Create file /etc/apache2/mods-available/proxy_http.conf and put this inside:

ProxyVia On
ProxyPreserveHost On
ProxyRequests Off

<Proxy *>
        Order deny,allow
        Allow from all
</Proxy>

Add these lines in /etc/apache2/apache2.conf to block paths that should not be visible

ProxyPass /.env !
ProxyPass /.git !

Enable these modules

a2enmod proxy
a2enmod proxy_http
a2enmod rewrite
a2enmod headers
a2enmod ssl
a2enmod remoteip

Optional: By default apache only looks at .conf files from /etc/apache2/sites-enabled for sites. If you want it to look at all files edit this line in /etc/apache2/apache2.conf

IncludeOptional sites-enabled/*

On busier proxies you can modify /etc/apache2/mods-enabled/mpm_event.conf for a better performance. Here is an example:

<IfModule mpm_event_module>
        StartServers                     5
        MinSpareThreads          25
        MaxSpareThreads          75
        ThreadLimit                      64
        ThreadsPerChild          25
        MaxRequestWorkers         250
        MaxConnectionsPerChild   0
</IfModule>

And restart apache2

systemctl restart apache2

Failover proxy

Create a user on failover-proxy with a private key and one on proxy with the public key that would sync the data from proxy to failover-proxy.

On failover-proxy create script /opt/bin/proxy_sync:

#!/bin/bash

#apache sync
/usr/bin/rsync -rl --safe-links --delete --rsync-path="/usr/bin/sudo /usr/bin/rsync" <user>@<proxy-ip>:/etc/apache2/ /etc/apache2/ 2>&1 >> /var/log/proxy_sync.log

#for letsencrypt certs
/usr/bin/rsync -rl --safe-links --delete --rsync-path="/usr/bin/sudo /usr/bin/rsync" <user>@<proxy-ip>:/etc/letsencrypt/ /etc/letsencrypt/ 2>&1 >> /var/log/proxy_sync.log

/usr/sbin/apache2ctl graceful

Make it executable

chmod +x /opt/bin/proxy_sync

Create a cron job for the script /etc/cron.d/proxy_sync:

*/15 * * * * <user> /opt/bin/proxy_sync

Keepalived

Keepalived is used to manage a floating IP.

To setup keepalived, install it on both servers:

sudo apt-get install keepalived

Copy the nagios check "check_http" to /usr/local/bin, from the /usr/lib/nagios/plugins of a server that has nagios-plugins installed (please don't install nagios-plugins on the reverse proxy, that package would install many dependencies).

Finally create the following /etc/keepalived/keepalived.conf on the master:

global_defs {
	notification_email {
		<email>
	}

	notification_email_from <email>
	smtp_server 127.0.0.1   
}

vrrp_script chk_apache {           	
        script "/usr/local/bin/check_http -H 127.0.0.1 -e 200"    	
        interval 3                      # check every 2 seconds
        weight 2                        # add 2 points of prio if OK
}

vrrp_instance floating_ip {
        interface ens3
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass SHAREDPASSWORD
        }
        virtual_ipaddress {
                <select-an-ip>
        }

        track_script {
            chk_apache
        }
}

Create exactly the same file on the failover proxy, just change priority from 101 to 100.

Explanations of the apache check

The track_script "chk_apache" uses nagios' script check_http to ask a webpage to the local apache every three seconds - and checks for a 200 "OK" error code. If the test succeeds, it adds 2 points to the priority of the server.

The server with the highest priority gets the IP address. In effect, we would have the master with a priority of 101+2, and the failover with a priority of 100+2.

If the check fails on the master, its priority goes down to 100, and the IP migrates to the failover, which still has a priority of 100+2.

Adding a vhost

In Apache we use vhost declarations to define each reverse proxy FQDN. In Ubuntu/Debian systems these are found in /etc/apache2/sites-available, typically one per vhost using a suitably descriptive name. They can also be wrapped into a single file, or of course into the main apache conf file. As they are effectively (includes) of the Apache conf, every change requires an Apache restart to apply:

apache2ctl restart

In the Debian/Ubuntu model you also need to enable a site one it's been defined, which is done with a link to the /etc/apache2/sites-available/ file newly created:

cd /etc/apache2/sites-enabled
ln -s ../sites-available/yournewvhost

This approach allows you to quickly and easily take a specific site offline if there's a problem, just by deleting the link in /etc/apache2/sites-enabled and restarting Apache.

Assuming your sites will be https from the proxy outwards, start with a 301 to force https:

<VirtualHost *:80>

       ServerName my.domain.name
       ServerAlias my.alias.domain

Redirect 301 / https://my.domain.name

       ProxyPass / http://my.realserver.nameorIP/
       ProxyPassReverse / http://my.realserve.nameorIP/

       CustomLog /var/log/apache2/my.domain.name.access.log combined
       ErrorLog /var/log/apache2/my.domain.name.error.log

</VirtualHost>

And then add an SSL entry

<VirtualHost *:443>
       ServerName my.domain.name

       SSLEngine on
       SSLCertificateFile /etc/apache2/ssl/mycertificate.crt
       SSLCertificateKeyFile /etc/apache2/ssl/mykey.key
       SSLCertificateChainFile /etc/apache2/ssl/myintermediatecertificateifneeded.crt

       ProxyPass / http://myrealserver.nameorIP/
       ProxyPassReverse / http://my.domain.name/
       ProxyPassReverse / http://myrealserver.nameorIP/

       CustomLog /var/log/apache2/my.domain.name.access.log combined
       ErrorLog /var/log/apache2/my.domain.name.error.log
</VirtualHost>

If you want the SSL proxy to also connect to the target as SSL, change the ProxyPass URLs appropriately and add to the vhost SSL (*:443) part

SSLProxyEngine On

If you run into issues with SSL connects to the target service with Apache log entries like

Error during SSL Handshake with remote server

then you may need additional settings in the vhost statement; try

SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off

and if that works selectively comment out to optimise (least settings for desired functionality).

Enabling SSL on a VM hosting a CMS behind a proxy

The CMS tries to detect if it's running on SSL automatically by checking some SERVER environment variables. When SSL is served from the proxy, the CMS can't detect it by normal means, so it needs a bit of help. In our case the solution is to edit the .htaccess file and add this directive:

<IfModule mod_env.c>
   SetEnv HTTPS on
</IfModule>

This was the fix for both Joomla and Wordpress VMs

Maintenance Page

If you are working on a website and want to show a maintenance page for everyone except you, this can be done from the Apache vhost on the proxy.

<VirtualHost *:443>
        ServerName my.website.com

        ProxyPass / http://ip.add.re.ss/
        ProxyPassReverse / http://ip.add.er.ss/

RewriteEngine On
RewriteCond %{REMOTE_ADDR} !^my\.ip\.addr\.ess
RewriteCond %{REQUEST_FILENAME} !/index.html
RewriteCond %{REQUEST_FILENAME} !/css/app.css
RewriteCond %{REQUEST_FILENAME} !/logo.png
RewriteRule ^.*$ /var/www/maintenance/index.html [L]
Header Set Cache-Control "max-age=0, no-store"

        CustomLog /var/log/apache2/my.website.com.access.log combined
        ErrorLog /var/log/apache2/my.website.com.error.log
</VirtualHost>

Redirect source IP to VMs

On the vm vhost add/edit the following lines:

SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!forwarded
CustomLog ${APACHE_LOG_DIR}/access.log forwarded env=forwarded

And enable 2 modules:

a2enmod remoteip && a2enmod headers

Redirect except one IP

Note: This works only for apache 2.4 onwards. For older versions of apache find something around RewriteEngine.

<If "%{REMOTE_ADDR} !='my.ip.add.re.ss'">
Redirect 301 / https://my.website.com
</If>

Get client's real IP address

It is a well known problem that when you run an apache web server behind a proxy, you cannot get client real IP address. To solve this issue we just have to run next commands on each backend server:

sudo a2enmod remoteip
sudo echo "RemoteIPHeader X-Forwarded-For" >> /etc/apache2/apache2.conf
sudo sed -i "s/LogFormat \"%v:%p %h/LogFormat \"%v:%p %a/g" /etc/apache2/apache2.conf
sudo sed -i "s/LogFormat \"%h/LogFormat \"%a/g" /etc/apache2/apache2.conf
sudo apache2ctl graceful

How to reverse proxy WebSockets

Enable the websockets module if not already

a2enmod proxy_wstunnel

Add these lines before the ProxyPass lines

        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*)           ws://172.17.16.17/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*)           http://172.17.16.17/$1 [P,L]