WordPress: Cannot connect to database – MySQL killed

/in , , /by

The MySQL server at the SysCrunch server was always being killed, the reason was due to lack of resources (RAM). Even after a memory increase to a total of 3+ GB, the problem persisted.

This blog post describes how to set up a monitoring tool (Monit) under CentOS 6 to monitor processes and to restart them when necessary. It also explains how to avoid the Apache server from exhausting the memory.

The problem

Essentially, the Apache server was consuming most of the RAM leaving little for other processes. Thus, MySQL was eventually killed as well as the ElasticSearch service. This happened as the number of web services and websites in the server increased. This would manifest when bots like googlebot started scanning intensively the server. Yes, Apache and MySQL are running in the same server with the current configuration.

Monit

The steps to install the monitoring tool Monit under CentOS 6 are similar to other service like Apache, you can also follow this tutorial or this one (Monit part) if you are under CentOS 7 and this might help for Ubuntu.

Install Monit

sudo yum install monit

Monit Web Interface

First open the configuration file, you can also use the nano editor.

sudo vi /etc/monit.conf

Find the set httpd port line, uncomment and update the configuration, here is a sample configuration:

set httpd port 2812 and
    use address 192.192.192.192  # Your IP address (private/public) for connections.
    allow 0.0.0.0/0.0.0.0        # What networks are allowed? Any network.
    allow admin:password         # Set username and password
    allow @monit # allow users of group 'monit' to connect
    allow @users readonly # allow users of group 'users' to connect readonly

Add monit to the system startup:

sudo chkconfig monit on

Start the monit service:

sudo service monit start

You can test the web interface in a browser using the provided IP address or hostname (e.g. http://192.192.192.192:8912 or http://hostname.com:8912). It will ask your for the username and password you have defined in the config file.

This is a sample capture of our server web interface:

Monit-web-interface-SysCrunch

Monit Monitoring

Monit can monitor multiple processes and execute actions based on some conditions like CPU and memory usage.

Apache Web Server

The paths might be different in your system, make sure to check them before. Find and uncomment the process httpd line, this is a sample config:

check process httpd with pidfile /var/run/httpd/httpd.pid
    start program = "/etc/init.d/httpd start" with timeout 60 seconds
    stop program  = "/etc/init.d/httpd stop"
    if cpu > 70% for 2 cycles then alert
    if cpu > 80% for 5 cycles then restart
    if totalmem > 1500.0 MB for 5 cycles then restart
    if children > 250 then restart

Essentially, the Apache server cannot take more than 1.5 GB, if it reaches that limit at least during 5 cycles, it gets restarted to free resources. The same happens if too many workers (250) are created. These values depend heavily on your system resources and the number of processes you are running.

MySQL Server

The default monit config file does not include a sample for MySQL. Following the Apache configuration and some samples from stack overflow, this is a proposed config that you can add after the Apache config:

check process mysql with pidfile /var/run/mysqld/mysqld.pid
  group mysql
  start program = "/etc/init.d/mysqld start"
  stop program = "/etc/init.d/mysqld stop"
  if failed host 127.0.0.1 port 3306 then alert
  if failed host 127.0.0.1 port 3306 then restart
  if 5 restarts within 5 cycles then timeout

If the server does not respond on port 3306, it raises an alert first and then does a restart. If too many restarts happen within a short period (5) then it timeouts.

ElasticSearch

Similar to the MySQL config, this is a proposed configuration that you can put after the Apache or MySQL configs.

check process elasticsearch with pidfile /var/run/elasticsearch/elasticsearch.pid
  start program = "/etc/init.d/elasticsearch start"
  stop program = "/etc/init.d/elasticsearch stop"
  if failed host 127.0.0.1 port 9200 then alert
  if failed host 127.0.0.1 port 9200 protocol http then restart
  if 5 restarts within 5 cycles then timeout

I haven’t seen any alerts or restarts of this service yet. Notice the protocol http condition for the restart, it might be necessary as ElasticSearch runs as a http application on that port.

Configuring Alerts

Monit lets you configure alerts by e-mail, in this case we use gmail as SMTP and you can also default to localhost to deliver the e-mails. You can refer to the Monit documentation for more details.

Find and uncomment the set mailserver line to ressemble to:

set mailserver smtp.gmail.com PORT 587 USERNAME "your-user@gmail.com" PASSWORD "your-password" using TLSV1 with timeout 30 seconds

You have to replace the username and password. You can then find the set alert line to define the recipient:

set alert dummy@syscrunch.com not on { instance, action }

This alert is skipping Monit actions like start, stop or performing user actions in order to avoid trivial cases.

IMPORTANT: Gmail will not send e-mails with this configuration unless you enable the less secured Apps option for your account at http://www.google.com/settings/security/lesssecureapps.

Apache Server RAM Solution

After monitoring the processes with Monit, the problem was with the Apache server that was running too many worker instances. This would lead to exhausting the system memory and lead to other processes being killed. The issue is that most of the web services (virtual hosts) were configured with a HTTP Keep Alive header, this allows for faster response times but it consumes more resources as the connection is kept open.

The solution was to remove the Header set Connection keep-alive from the virtual hosts. It can also be set explicitly to close Header set Connection close. Alternatively there are other options to tune the persistent connections like MaxKeepAliveRequests  and KeepAliveTimeout , it’s recommended to have a good understanding since for HTTP 1.1 clients, the default option is to keep connections alive, you can read more in the Apache docs for Keep Alive.

In the end with this configuration, it was possible to launch 3000 thousand requests with 60 concurrent requests using Apache Benchmark (3 parallel calls) and the server handled it without any problems. Here is a sample output of one of the tests:

Server Hostname:        www.syscrunch.com
Server Port:            80

Document Path:          /
Document Length:        47998 bytes

Concurrency Level:      20
Time taken for tests:   122.177 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      48396000 bytes
HTML transferred:       47998000 bytes
Requests per second:    8.18 [#/sec] (mean)
Time per request:       2443.541 [ms] (mean)
Time per request:       122.177 [ms] (mean, across all concurrent requests)
Transfer rate:          386.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       55  139  93.7    113     973
Processing:   320 2278 594.2   2249    8164
Waiting:      188 2011 601.7   1987    8009
Total:        380 2417 594.7   2375    8336

Percentage of the requests served within a certain time (ms)
  50%   2375
  66%   2481
  75%   2562
  80%   2616
  90%   2855
  95%   3265
  98%   4264
  99%   4564
 100%   8336 (longest request)

I wrote this article to document the solution to this problem and it might be updated over time.

Hope it’s useful for some other people.

Tired of Firefox fast releases?

/in , , /by

Firefox has been releasing major versions every 6 weeks since April 2011 going from a version 4 to version 20 (time of writing). I want to present some numbers in terms of versions and browser market share of what happened since then.

What was the goal of the fast releases? Wikipedia quotes:

“The stated aim of this faster-paced process is to get new features to users faster”

The features have been added but in the busy life of users and workers it’s not the priority, we just need a browser that works, gets faster and doesn’t get in the way. I’m speaking as a user and also as a developer who loved Firefox in the past. Honestly, from a user perspective it does not appear as loads of features. Back in 2011, there were lots of articles explaining the move and possible issues, good analysis and thoughts and user concerns.

Let’s get into the numbers. I prepared a graph that shows the version evolution over time, it uses the data form the History of Firefox in Wikipedia.


Observing at the version release, one can clearly see that the release-pace is now very fast, specially compared to previous releases. In despite of this effort the market share continued decreasing. See below the market share graph, the data is from StatCounter aggregated by Wikipedia (desktop and mobile combined).

Firefox and Internet Explorer lost market share to Chrome. However IE was loosing faster till early mid 2012 where the decreasing pace of Firefox and IE is similar. What is interesting is to see the lines of Firefox and Chrome, is almost the opposite since early 2011, when the fast-releases started.

Maybe, you’re wondering what happened before 2011. The following graph is a plot of browser market share since 2007, the data used is from W3Counter.

One can clearly see that the browser that lost the most is Internet Explorer. Firefox is yet decreasing but in Mid 2010 had more than 30% market share. The 10% lost from then up to now looks steady (decreasing) and there is no any hint that fast-releases helped or affected Firefox.

What would it take for users to get excited about Firefox (again)? These are my thoughts:

    1. Silent upgrades. Do not ask to press a button to upgrade every 6 weeks. There are other means to catch user attention and hence loyalty, opening a tab after an upgrade as Firefox does, it’s a good one. Or even to put the version number in clear sight in the User interface.
    2. Javascript performance, this would definitely be a killer, a reason why currently Chrome is my browser choice. This seems to be coming with OdinMonkey (only 2 times slower than native code) in Firefox 22, in june 2013.

What about you? Do you use Chrome, Firefox, both?

Why do we need Flash?

/1 Comment/in , /by

Flash is a technology that most people surfing the web is aware and used to, at least 99% PC-users have installed the Flash Player runtime according to Adobe by July 2011. Although those numbers are from a single source and it happens to be the Company behind it itself, it is possible to say they are credible enough and some other sites provide similar statistics too (e.g. RiaStats).

The applications for Flash have changed over time, in the beginning developers were using it for animations and banners all the way to play online video and all the major online video sites still use it widely, namely YouTube, Yahoo Video, Bing Video, Daily Motion, Vimeo and others. Most developers will agree that the best use of Flash is for online video and that’s thanks to its streaming capabilities using the Real Time Messaging Protocol (RTMP). However since the era of HTML5 which can embed videos using the video tag, some players (e.g. YouTube HTML5 Player) have migrated or offered both alternatives. However there is more, Flash has gone far beyond after the Rich Internet Applications (RIA) offering development environments and APIs such as Flex Builder based on the Flex SDK.

Thanks to all the tools and resources available to build flash content, it is possible to achieve stunning visual results. However, and most authoring people will agree, it is necessary to build flash content carefully. That means, to understand the basis of the Flash Players and how the content is executed. Developers should not forget that the code is executed on the client side since the players are just plugins executed inside the web browsers and that the computer memory and processor will be taxed during the execution. There is nothing more disturbing that being surfing quietly on the web and to land on a site that has flash content that makes your computer over-boost and give you the feeling that your computer is trying to take-off. It is not completely the problem of the technology used to build flash content or execute it (players) but the way the content has been produced/written. As usual, latent problems open the doors for new opportunities and solutions and this time has not been an exception. Some decided to migrate to HTML(5)/CSS(3) and AJAX, even of the pain that causes to make it cross-browser compatible and some others have gone even wilder developing browser plugins to block flash content such as FlashBlock for Chrome or FlashBlock for Firefox which are not just used by few but millions all over the internet.

So, Why do we need Flash? We need it to run the existent content that has been authored for flash. Although lots of sites have migrated or created a non flash version of their content to allow devices which do not support flash to visualize the content and those are mainly iOS devices (iPad, iPhone, iPod) and devices with small screens. But, is it possible to achieve the same results as in Flash using HTML5/CSS/Javascript? It depends, if you’re just after visually good looking sites or Apps with animations and effects, yes, CSS and Javascript can do that for you. However if you are writing RIA applications, it depends on what you want to leverage, your skills and background; you have plenty of frameworks, libraries and components available for Flash (AS2/AS3) as well as for Javascript (jQuery based, MooTools, Node.js). Personally, I would go on the side of the standards (HTML5/CSS3/Javascript). after all, Adobe is one of the active supporters of HTML5.

So, What do you need Flash for?