MySQL Database Design and Tuning

by Robert D Schneider

Overview

The authoritative, hands-on guide to advanced MySQL programming and administration techniques for high performance is here. MySQL Database Design and Tuning is the only guide with coverage of both the basics and advanced topics, including reliability, performance, optimization and tuning for MySQL. This clear, concise and unique source for the most reliable MySQL performance information will show you how to:

• Deploy the right MySQL product for your performance needs.
• Set up a performance management and monitoring environment using tools from MySQL.
• Implement the right indexing strategy
• Apply good performance strategy when developing software to work with the MySQL database.
• Configure dozens of variable to correctly tune the MySQL engine.

If you deal with the intricacies and challenges of advanced MySQL functionality on a daily basis, you will be able to build on your knowledge with author Robert Schneider's real-world experiences in MySQL Database Design and Tuning.

Product Details

ISBN-13:	9780672327650
Publisher:	MySQL Press
Publication date:	06/27/2005
Series:	Developer's Library Series
Pages:	432
Product dimensions:	6.90(w) x 9.10(h) x 1.00(d)

About the Author

MySQL® Database Design and Tuning About the Author

Robert D. Schneider has more than 15 years of experience developing and delivering sophisticated software solutions worldwide. He has provided database optimization, distributed computing, and other technical expertise to a wide variety of enterprises in the financial, technology, and government sectors. Clients have included Chase Manhattan Bank, VISA, HP, SWIFT, and the governments of the United States, Brazil, and Malaysia.

He is the author of Optimizing Informix Applications and Microsoft SQL Server: Planning and Building a High Performance Database. He has also written numerous articles on technical and professional services topics. He can be reached at [e-mail].

Table of Contents

The Purpose of This Book.

Who Should Read This Book?

Benefits of Using This Book

How to Use This Book

Information About MySQL Software and Environments

Information About Examples

I. INTRODUCTION.

1. Setting Up an Optimization Environment.

Pretesting Preparation

Hardware

Connectivity

Software

Data

Your Test Plan

Change One Variable at a Time

Organizational Support

After Testing Is Under Way

Posttesting Implementation

Recording Your Results

Making Improvements

2. Performance Monitoring Options.

MySQL Tools and Utilities

Character-Based Tools and Utilities

Graphical Tools

Logging

Operating System Tools

Windows

Linux/Unix

3. MySQL Overview and New Performance-Related Features.

MySQL Products

MySQL Core Technologies

Database Storage Engines and Table Types

Distributed Computing Technologies

Graphical Tools and Assistants

Connectors

APIs

Utilities

Performance-Related Tools

About the Versions Used for This Book

MySQL Versions

Operating Systems

Performance-Boosting Features from Version 4.0 Onward

II. ADVANCED DATABASE DESIGN CONCEPTS.

4. Designing for Speed.

Choosing the Right Storage Engine and Table Type

InnoDB

MyISAM

MEMORY

MERGE

CSV

ARCHIVE

BerkeleyDB (BDB)

MaxDB

NDB

FEDERATED

Optimizing Table Structure

Specifying Row Format

Specifying Table Size

Index Key Compression

Checksum Integrity Management

Column Types and Performance

Using Views to Boost Performance

Reduced Returned Columns

Reduced Returned Rows

Reduced Inefficient Joins

Simpler Updating with Integrity Checking

When Not to Normalize

5. Using Constraints to Improve Performance.

UNIQUE Constraints

PRIMARY KEY

FOREIGN KEY

DEFAULT and NOT NULL

ENUM

SET

6. Understanding the MySQL Optimizer.

What Is an Optimizer?

Optimizer Diagnostics and Tuning

The ANALYZE TABLE Command

The OPTIMIZE TABLE Command

The EXPLAIN Command

Helping the Optimizer

7. Indexing Strategies.

Index Key Terms and Concepts

Index Reports and Utilities

SHOW INDEX

Assisting the Optimizer with Indexes

Index-Related Logging

MyISAM Indexing Features

Key Cache

The myisamchk Utility

Index and Table Compression

InnoDB Indexing Features

Index Structure and Storage

Buffer Pool

Memory Pool

Adaptive Hash Index

Automatic Foreign Key Index Generation

Indexing Scenarios

PRIMARY KEY

Filter Columns

Join Columns

Index Cardinality

Character Versus Numeric Indexes

Multicolumn Indexes

Partial Indexes

Ascending Versus Descending Indexes

Storing Tables in Column-Sorted Order

III. OPTIMIZING APPLICATION CODE.

8. Advanced SQL Tips.

Improving Searches

Leveraging Internal Engine Caches

Controlling Data Retrieval

Reducing Security Costs

Off-loading Processing Work

Boosting Join Performance

Substring Searches

Improving Temporary Table Performance

Managing View Performance

Subqueries

Using Math Within SQL

UNIONs

Sorting

HANDLER

Improving Data Modification

Improving INSERT

Improving DELETE

9. Developing High-Speed Applications.

Understanding Locking

General Locking Overview

InnoDB Locking Overview

Optimal Transactions

Key Transaction Terms and Concepts

Cost of Transactions

Transaction Performance Suggestions

Stored Procedures and Functions

Difference Between Stored Procedures and Functions

When to Use Stored Procedures or Functions

When Not to Use Stored Procedures or Functions

Stored Procedure Optimization

Triggers

Current Trigger Limitations

When to Use Triggers

When Not to Use Triggers

Writing Efficient Application Code

General Suggestions

IV. OPTIMIZING AND TUNING THE MYSQL ENGINE.

10. General Server Performance Parameters and Tuning.

Server Settings Scope

Connectivity

Creating a Connection

Managing a Connection

Memory Management

Locking mysqld in Memory

Thread Memory Settings

MEMORY Tables

Caching

Application Control

Resources

Optimizer Control

Concurrency

User Resource Control

11. MyISAM Performance Enhancement.

Optimal Data Storage

Table Reporting and Control

Table Maintenance and Repair

Controlling Sorting Resources

MyISAM and Memory

The Key Cache

Speeding MyISAM Operations

Loading Information

Improving FULLTEXT Searches

Concurrency Considerations

12. InnoDB Performance Enhancement.

InnoDB Storage Architecture and I/O

Indexes

Pages

Extents and Segments

Tablespaces

Built-in Performance Enhancers

Data Storage

Log Files

InnoDB and Memory

Buffer Pool

InnoDB Operational Control

Thread Performance

Improving Concurrency

Improving Large-Scale Operations

Speeding Up Shutdown

13. Improving Disk Speed.

General Suggestions

Deploying the Fastest Possible Disk Drives

Configuring Memory Caches Correctly

Implementing a RAID Strategy

Distributing Temporary Directories

MyISAM Suggestions

Symbolic Links

Table Compression

InnoDB Suggestions

Choosing the Right autoextend Setting

Using Raw Devices

Alternative Disk Writing Methods

Table Defragmentation

Operating System—Specific Disk Suggestions

Linux/Unix Suggestions

Windows Suggestions

14. Operating System, Web Server, and Connectivity Tuning.

Operating System Considerations

Assumptions

Linux/Unix

Windows

Web Server Performance Considerations

Choosing the Right Server Topology

General PHP Suggestions

Apache/PHP Suggestions

Internet Information Services (IIS)/PHP Tips

Connectivity Tuning

Protocol Choices

Costs of Hostname Resolution

Costs of Creating and Destroying Connections

Costs of SSL

15. Improving Import and Export Operations.

Speeding Data Exports

The Importance of Regular Exports

Avoiding Excessive Memory Consumption

Concurrency and Exports

Retrieving Subsets of Information

Copying Table Structures Without Data

Delaying Index Re-creation–MyISAM

Delaying Index Re-creation–InnoDB

Preparing for Data Reload

Accelerating Data Loading

Managing Concurrency

Handling Errors

Engine-Specific Tips

V. DISTRIBUTED COMPUTING.

16. Optimal Replication.

Introduction

How Does Replication Work?

Is Replication Right for You?

Creating a Replication Strategy

High-Performance Replication

Network Considerations

Master Server Performance Considerations

Implementing Best Tuning Practices

Managing Slave Resources

Slave Server Performance Considerations

17. Optimal Clustering.

Introduction

How Does Clustering Work?

Nodes

Shared-Nothing

Cluster Clients

Storage Engine

Transporter

Data Distribution

Checkpoints

Is Clustering Right for You?

Clustering Versus Replication

Creating a Clustering Strategy

Choosing the Right Version

Cluster Topology

Configuring MySQL Cluster for High Performance

General Clustering Best Practices

Network Considerations

Transports

Management Node Considerations

SQL Node Considerations

Data Node Considerations

VI. CASE STUDIES.

18. Case Study: High-Hat Delivers!

Problem Queries

Package Status Lookup

Shipping Option Lookup

Random Transaction Bottlenecks

Diagnosis

Solution

Implementing These Solutions

19. Case Study: Friends Fly Free-for-All–A Promotion Gone Wrong.

Server Availability

Diagnosis

Solution

Application and Transaction Issues

Diagnosis

Solution

20. Case Study 3: Practice Makes Perfect.

Data Importing

Diagnosis

Solution

Clustering

Diagnosis

Solution

Stored Procedures

Diagnosis

Solution

Index.

Preface

Introduction

The Purpose of This Book

MySQL is the foundation of many of today's highest quality, most robust database-driven solutions. However, users continually grow more demanding: A rich application that runs sluggishly is unacceptable, no matter what functionality it offers. Unfortunately, the cost of job cutbacks and outsourcing means that most overloaded IT professionals barely have enough time to deliver a minimal solution, much less master the minutiae of their database engine, which, in the case of MySQL, grows richer and more complex with each release.

Performance often takes a backseat to merely hitting the schedule. "Just ship it now and make it faster later" is a refrain heard around the world today. Regrettably, "later" rarely arrives, and designers, developers, and administrators end up delivering suboptimal solutions and then bracing themselves for the inevitable complaints from users. In many cases, these problems would never have come to pass if the overloaded designers and developers had enough time to fully research and exploit the numerous avenues to boost MySQL response. This book was written to help these beleaguered professionals jump-start their knowledge of MySQL performance tuning, letting them design, develop, and maintain the fastest MySQL solutions in the shortest amount of time.

Who Should Read This Book?

MySQL Database Design and Tuning is aimed at busy database designers, database and system administrators, webmasters, and developers who want to build, deploy, and manage the fastest possible solutions based on MySQL.

To take full advantage of this book, application developers should understand the following:

• Standard Structured Query Language (SQL)—Because SQL is the underpinning of MySQL database access, it's important that you at least comprehend basic SQL (simple queries, INSERT/UPDATE/DELETE operations, and so on).

• A programming language such as Java, C++, and so on—SQL is often paired with a procedural programming or scripting language to create MySQL-based applications. Although MySQL Performance Optimization is not a programming guide, it does discuss the performance interplay between the programming language and the MySQL engine.

Readers who are responsible for designing and/or administering databases should understand the following:

• Basic relational database design theory—To make the most of this book, you should be aware of how to design a simple database, as well as understand data normalization, constraints, and so on.

• Intermediate or advanced SQL—As a database administrator/analyst (DBA), you are often called on to produce more sophisticated SQL on behalf of your users. MySQL Database Design and Tuning will help you improve the responsiveness of your more complex join, subquery, index construction, and transactional challenges.

• MySQL engine architecture—This book does not expect you to have downloaded the MySQL source code or to construct hashing algorithms in your spare time. However, if you are familiar with the elegant MySQL product line architecture, you are likely to make better use of this book's recommendations.

System and web administrators should be familiar with the following:

• MySQL's engine architecture—Database performance issues are often misdiagnosed as hardware and/or operating system flaws. By understanding the components that make up the MySQL database, it's more likely that you will be able to accurately identify and fix (or even prevent) database response concerns.

• Configuring server operating systems such as Windows and Linux—There is interplay between the MySQL database and the underlying operating system. Fortunately, you can employ a number of operating system–specific settings to make your operating system more MySQL-friendly.

Finally, some words for the open source developer. Unlike any other widely available database, MySQL's open source architecture, customer-friendly licensing, and source code availability means that you can programmatically change the behavior of the engine itself. This book, however, focuses on using the commercially available versions of MySQL; this book does not expect any reader to have the time, skill, or inclination to alter MySQL's source code.

Benefits of Using This Book

Readers of MySQL Database Design and Tuning will help themselves and their organizations in at least five ways:

• Simplicity—There's a common belief that the simplest solution is by definition a slow solution. However, this is often incorrect. In fact, a well-designed, optimized database application is frequently cleaner and more elegant than an overengineered product. Good performance and simple design are rarely mutually exclusive. This book always opts for the simplest, fastest solution.

• Productivity—Database-centric applications often serve as the cornerstone of an organization's operations. A nonoptimized solution wastes each user's time. When multiplied by the large number of internal and external users affected by these delays, the cost to the organization's productivity can be staggering. MySQL Performance Optimization is filled with easily implemented tips and tricks that will have a dramatic, positive impact on performance.

• Reliability—Response and reliability problems often are mistaken for each other. For example, imagine that you are a user of a popular website that is built on MySQL. You have just submitted an e-commerce payment form. Unfortunately, the site database and application designers never planned for large numbers of users inserting information at the same time, so your browser appears to "hang" while waiting for a response. From the user's perspective, is this a flaw in response or reliability? It really doesn't matter because you now have an unhappy customer who thinks that your website is "buggy." Faster systems equal more reliable systems.

• Cost control—Hardware prices continue to fall; memory and disk drives are now so cheap that you might wonder if they will be offered in cereal boxes before too long. Nevertheless, many organizations that employ MySQL are cost conscious. Squeezing an extra 10% or 20% from your applications might spell the difference between your raise money being spent on you or on a new CPU.

• User satisfaction—Although your users will likely love your MySQL-based solutions, it's doubtful that they will want to spend any more time staring at their screens awaiting response than is absolutely necessary. Each year, users grow more sophisticated and less tolerant of delay. In some cases, a perfectly good system is abandoned because of performance problems. MySQL Performance Optimization is designed to help you delight your users by delivering the fastest, most reliable solutions.

How to Use This Book

This book is divided into six major sections:

• Part I: Introduction—This section describes how to get the most from this book, its structure and examples, and MySQL's architecture, as well as how to set up an optimization environment.

• Part II: Advanced Database Design Concepts—Your MySQL solution is only as solid and responsive as the database design upon which it rests. This section explores how to choose the correct options when designing your tables, the impact of constraints on response, the excellent diagnostics offered by the MySQL optimizer, and how to leverage indexes for exceptional performance.

• Part III: Optimizing Application Code—Your applications will likely interact with MySQL through one or more programming/scripting languages, even if it's just SQL. For that reason, this section examines how to write the most efficient SQL, as well as special considerations when using popular programming technologies, such as Java, C, ODBC, Perl, and so on. Because newer versions of MySQL offer stored procedures and triggers, this section also looks at how to make these exciting new technologies as speedy as possible.

• Part IV: Optimizing and Tuning the MySQL Engine—Administrators have dozens of choices to make when setting up and maintaining their MySQL environments. These choices range from engine and operating system parameters to disk management and bulk data operations. This section is devoted to helping these administrators make intelligent decisions when confronted with these situations.

• Part V: Distributed Computing—Replication and clustering are two key MySQL technologies for spreading your processing load among multiple computers. This section of the book examines when to deploy these powerful capabilities, as well as how to tune them for optimal performance.

• Part VI: Case Studies—The final section of this book ties the information found in the earlier chapters together in a series of multidimensional, real-world case studies.

Whether you choose to read all sections will likely be determined by your MySQL-oriented responsibility. For example, if you are an application developer, it's likely that you'll focus on Parts I, II, III, and V; you are probably not going to spend much time on Part IV's engine configuration. However, if you are tasked with tuning the database engine or the operating system, Part IV will be of supreme interest to you. In any case, the optimization techniques and examples have been purposely designed to be as independent of each other as possible.

Information About MySQL Software and Environments

The frequency with which MySQL AB releases new versions and products can be overwhelming. This book uses products from both the 4.1 series (versions 4.1.7 through 4.1.10), along with newer versions, from 5.0 through 5.0.3. When applicable, the performance-specific features of the 5.1 release are discussed. This book also highlights some of the excellent new graphical products, such as the MySQL Administrator and MySQL Query Browser.

MySQL's open source architecture means that the product runs on a tremendous variety of hardware and operating system environments. However, aside from the sections in which operating system tuning in a MySQL environment is discussed, this book always strives to make the examples as platform-independent as possible.

Information About Examples

Founded during the height of the dot-com boom in 1999, High-Hat Airways has grown into the world's largest airline. Its unique blend of hip, in-your-face marketing, low prices, and creative corporate accounting helped propel the airline past sluggish industry dinosaurs.

Its fleet has expanded from one 20-year-old rented Boeing 737 to more than 550 planes. During the next year, more than 275 million customers will board a High-Hat flight and be greeted by the familiar "Go sit down over there!" from a friendly flight attendant.

The year 2002 saw the indictments of the CEO and CFO, along with the mysterious disappearance of four members of the Board of Directors. However, these setbacks proved to be mere road bumps on the path to success: The value of High-Hat's stock has grown thirtyfold in less than three years.

As part of its ongoing commitment to cost control, High-Hat Airways has decided to convert all internal systems to open source. Naturally, MySQL will be a cornerstone of this new philosophy. However, the already overburdened IT staff has a monumental task ahead: to consolidate 27 internal applications running on 11 different database platforms. The most important applications are as follows:

• Reservations

• Flight logistics and scheduling

• Catering

• Luggage management

• Loyalty programs

To make matters worse, the CIO has promised both internal users and the general public that High-Hat's systems will have the fastest response time in the industry. The CIO is also refereeing a religious war between Microsoft zealots and Linux enthusiasts. The result is that the new solutions will be evenly split between the two operating systems.

As you proceed through this book, you'll notice it continually refers to the challenges faced by the High-Hat Airways IT staff as they struggle to deliver quality, high-performance solutions based on MySQL. Luckily, very simple, easily implemented changes can result in massive speed gains, and that is what this book plans to help you achieve.

To keep things simple, this book introduces High-Hat's new applications, database structures, and engine configurations on an as-needed basis, in concert with the specific performance concept being examined at the time. This means that you can focus on MySQL optimization, rather than becoming an expert in the internal systems of High-Hat Airways. Finally, the database design, SQL, and code examples are always as simple and to-the-point as possible while still conveying the appropriate performance messages.

© Copyright Pearson Education. All rights reserved.