WordPress database slows down with lots of pages


Since I’ve been doning most of my customers’ sites in WordPress lately, I thought I’d write about some of the odd and difficult to track down WordPress issues I’ve dealt with.

This post is about a site I’ve been working on for about a year now: VaultDenim.com.

This company has been growing at extraordinary rates lately. They are a direct sales company specializing in selling designer jeans at reduced prices. In addition to the custom programming I’ve done to facilitate their business needs, I initially set the site up on WordPress.

They use WordPress to handle the logins and other things that WordPress does well. I added a feature where each new consultant gets a vanity URL when they sign up. I do this by adding a post to the wp_posts table with the necessary credentials to make it work as a vanity URL: set it as a page, set it to use the template I set up, put the post name as their name so that you can go to vaultdenim.com/name to see their personalized page. This is automated with a desktop application that accesses the database directly and not through the WordPress scripts. This has resulted in a WordPress installation with several thousand pages and a few hundred posts.

I have a pretty standard process for setting up a new WordPress installation. I put the theme in, the plugins I like to use, set the permalinks, set up the categories to facilitate WordPress operating as a CMS as opposed to a blog.

I typically set the permalinks to be category/postname.

Recently, we experienced a pretty serious issue with the server crashing almost daily. We knew that the cause of this was open Apache connections.

In troubleshooting the cause of this, our host wasn’t very helpful even though we were on a dedicated server. They wouldn’t all us root access to the box. We had to call them to get MySQL logs. They didn’t offer a command line restart option. It was a mess. We examined every application, database connection, and external program that accessed the server. They could not tell us what scripts or services were causing this.

We looked into DOS attacks, persistant MySQL connections, external desktop applications that accessed the database, WordPress, caching, plugins, bad code all to no avail.

We eventually were able to isolate the issue as being too many sleeping Apache connections to MySQL. However, we were unable to determine what script/plugin/application/user was causing these persistant connections. The server would get MySQL sleeping connections that would sit idle of 600 seconds until they timed out. They would just build and build until the server died.

The site was going down daily for over a week before we found the cause of this issue.

In the WordPress database, I found an entry in the wp_options table called rewrite_rules that was over 3MB and almost 3,000,000 characters. I knew this field was used to handle the permalink rewrites as when I added a new page to the database directly, it wouldn’t render in the browser unless I ran wp_flush(). This function processes the rewrites.

Eventually, we found an article that helped us with this exact issue found here. This artile basically pointed out that if the permalink structure starts with string, WordPress has to keep track of all of the page rewrites because it can’t tell the difference between a page and post since they are both varchar strings and there is no other way to differentiate between the two.

By changing the permalink structure to a standard yyyy/mm/dd/postname structure, the rewrite_rules field was reduced to a more manageable size. The server has yet to go down since this.

Apparently, according to that article having the permalink structure start with a number, WordPress doesn’t need to store all of the rewrite rules for the pages since it can tell that any permalink that starts with string is a page.

I hope this helps someone save some time. But like always, feel free to hit me up if you need help.