Pro-tips for production server setup (when shit hits the roof)

This post is not related to production deployment automation. This post is our voice of experience (read as: major issues caused due by “minor” missing server configurations) and how we resolved them.

These are changes that would help you in case you are dealing with

  • Upgrading your server stack that uses HAProxy and SSL (We were using EngineYard)!
  • Nginx configurations to prevent spoofing (and a rant on Cloudflare Always Online crappy product).

Upgrading your server stack

For the past few years, we had been using m1.large AWS EC2 servers from EngineYard. When AWS introduced better pricing for the newer servers, we migrated to the new servers. We setup the new server and migrated our old IP address to the new server. Obviously, there was no need to make any changes in the DNS. So, we thought all was well until our customer started complaining!

The portal administrator required to know the remote IP address as an additional level of logging and security and all of a sudden (from the date we migrated to the new servers), the remote IP was always the private IP address of the EC2 instance!

After contacting EY support, we had a few rounds of barking up the wrong tree. The following stunts failed but I am mentioning it here so that you know you’re in going in the wrong direction!

Stunt 1: CNAME, ELB and HAProxy games

Initially it was assumed that HAProxy handles remote IPs incorrectly when SSL is involved and we we need to use an Elastic Load Balancer or setup stunnel! Or disable haproxy (not recommended though). We followed instructions mentioned at and setup and “App terminated” ELB. Then we added the CNAME in our DNS.

No luck!

Stunt 2: Route53 hiccup

Since we had initially setup an A record with the Elastic IP, it was deemed that the ELB name entry was being by-passed and hence things were not working. So, it was recommended to move the domain to Route53 instead of our current Registrar.

Now this was suddenly a big issue with our customer and just when there was enough shit hitting the roof, EY support had another idea that worked! There was no need to change the A record or add the CNAME entry.

Here is a concise steps of changes that are required:

  • Add an ELB (Elastic Load Balancer)
  • Set the SSL terminated option to “ELB terminated” (not App Server terminated!)
  • Copy the same SSL certificate that’s on the app server to the ELB.
  • Change the nginx configuration to listen on port 81 instead of 80 (http) and 444 instead of 443 (https)

Now, HTTP(80) and HTTPS(443) requests are handled by the ELB (since our app is listening on 81 and 444). It then forwards the request with the correct “remote IP” to our application server and things are finally back to normal. Remember that the SSL certificate is for one domain only. Since we had it for http://www.ourdomain.com, we had to setup nginx 301 permanent redirects to the www domain.

Lessons learnt – when upgrading hardware resources, certain security policies change and can have adverse impact

Prevent Spoofing with proper nginx configuration

It so happened that our customer complained about Google Search results going wrong! As the owner of a site called fundamentor.com, when he google’d for it, very surprisingly, another site came up too!

spoofed site 2

When we actually clicked on that link – we were very surprised to see exactly our site available there. Thinking that there may be someone who has copied our HTML and updated it, I tried to login and I was able to login too!. Now we were worried and shit hit the roof. As you can see in the image below, the URL is for superpromoapp.com but its our web-portal that is displayed and fully functional. Had our servers been hacked? Had our source code been compromised?

spoofed site 4

When we checked out access.log file in nginx and found (to our relief) that we had been getting requests to http://superpromoapp.com and they were being served! On further investigation, we realized that our nginx configuration has only one server block. It’s mentioned in the documentation of nginx listen directive that

If none of the directives have the default_server parameter then the first server with the address:port pair will be the default server for this pair.

That’s exactly what was happening. So, superpromoapp.com has a malicious HTTP server that was rewriting the URL to hit the real server (fundamentor.com) and acting as a pass-through. Luckily we had found this early on otherwise all data including sensitive information would pass through this middle-man. They even had their own robot.txt configured to ensure good responses to crawlers and get a better ranking! (I still don’t know what they would get out of a better ranking) but to prevent this unwanted access, we then added the following to our nginx configuration.

server {
  listen 80;
  server_name fundamentor.com www.fundamentor.com;
  ...
}

# This was added to prevent any other domain accessing our site.
server {
  listen 80;
  server _ default_server;
  return 520;
}

Now, when we tried to open http://superpromoapp.com we realized that the spoofed site had cached all the content via “CloudFlare Always On” technology. Since this product acts as a reverse proxy, you can get content from virtually any site if you configure your DNS. When we reported abuse, they said that this was a hosting issue and diverted our query. Probably unrelated but worth a read about CloudFlare crappy support ToS!

spoofed site 1

When a day later, the cache was invalidated, the requests to http://superpromoapp.com were rejected by nginx with Error code 520 (Origin Error) and we finally resolved the problem ourselves!

spoofed site 3

Lesson learnt – Among other directives, don’t forget to add an nginx default_server block.

Hope these minor server configurations help you ensure that hardware upgrades or nginx server configurations allow you to have a better experience than we faced!

Advertisements

About Gautam Rege

Rubyist, Entrepreneur and co-founder of Josh-Software - one of the leading Ruby development shops in India.
This entry was posted in Case Studies and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s