Testing MySQL repository packages: how we make sure they work for you

Around nine months ago, we launched the MySQL yum repositories for Enterprise Linux and Fedora, followed by apt repos for Debian and Ubuntu back in May. We’re extremely happy that the repos have proved to be a big hit with the community: the monthly number of downloads hit 100K recently and it is still growing briskly. Indeed, the repos are now our most important community distribution channel for the MySQL product family on these platforms.

Right from the outset, it was clear that a thorough QA regime would be very important. The yum and apt projects involved a complete rearchitecting of our rpm and deb packaging, we expected the repos to become very popular, and not least: a key aspect of the repo distribution model is immediacy. If you release broken packages through your repos, lots of users are going to be hit by problems very quickly. So we made very sure that QA was placed front and center of the repo projects.

In this posting, we will describe the way we currently test MySQL packages that go into the repos. We will focus on the newest MySQL releases (5.6.20 and 5.5.39, both coming very shortly), and the most recently added platform, namely Enterprise Linux 7. The principles and overall approach are very similar between the yum and apt repos, while the specific test scope and types of install scenarios we test vary between different MySQL versions and OS flavors.

Objectives and strategy

Boiling down the motivation for our repo QA efforts to a few bullet points, this is what we get:

  • Provide users with overall high quality packages
  • MySQL is a core system component: our packages must not break the system in any way
  • Make sure we completely replace the distro native “mysql” without breaking compatibility
  • Make sure that distro native third party applications that depend on MySQL continue to work properly
  • Build and maintain user confidence in the official MySQL repos

The repository testing takes as its input a set of packages that comes from our continuous integration build and test system. The main focus is on testing the packaging, as opposed to the package contents; most of the testing of the product itself happens in a separate, parallel QA process. The entire process is applied for each and every maintenance release of MySQL.

The overall scope of testing covers numerous scenarios involving installation and upgrade, including verification of the resulting installs, and testing to make sure that third party applications that depend on MySQL work as expected. This is also the point where our chosen example becomes particularly interesting: as we have highlighted previously, Red Hat chose to ship an older, forked version of MySQL natively in EL7, and this meant we had to add some new testing scenarios in this latest round in order to make sure we got things right.

Scope and approach

Here is an overview of the different categories of repository package testing we do and the scenarios we cover, specifically for MySQL on EL7:

Installation testing

  • We install the following mysql community packages: mysql-community-bench, mysql-community-client, mysql-community-common, mysql-community-devel, mysql-community-embedded, mysql-community-embedded-devel, mysql-community-libs, mysql-community-server, mysql-community-test packages, all at once.
  • After installation, we validate that we have a running service, we connect to it, we verify plugins, do version checks and check mysql system tables, information_schema tables etc. in order to ensure that we have a valid install.
  • We test multilib support to ensure that users can run 32 bit applications with 64 bit MySQL packages.
  • We do individual package installation in order to ensure that the correct dependencies get installed automatically. E.g. the mysql-community-server package depends on mysql-community-client, mysql-community-common and mysql-community-libs.

Upgrade testing

As part of upgrade testing, we cover a number of different upgrade flows: upgrades between different server releases from the MySQL repo; upgrade from distro native packages to MySQL repo packages; upgrade from older MySQL repo packages to new ones, and finally that none of these three upgrade types break multilib support.

Let us take a more detailed look at each of these flows:

Upgrades between MySQL release series in the MySQL repo

As we have multiple MySQL major release series (5.5, 5.6 and 5.7) in the repository, we cover relevant upgrade flows between these releases in order to ensure that all the packages we deliver are compatible with each other:

mysql55-community → mysql56-community
mysql56-community → mysql57-community
mysql55-community → mysql57-community

Upgrade from distro native to MySQL repo with different MySQL versions

Here we test upgrade or replacement of native distro packages with real MySQL packages in order to ensure that nothing breaks. Example: EL7 comes with MariaDB 5.5.35 by default, we want to offer users a move to the real MySQL from MariaDB without any compatibility issues, so we thoroughly test upgrades from the distro native MariaDB to the different MySQL versions that we offer from the MySQL repo:

mariadb5.5.35 → mysql55-community
mariadb5.5.35 → mysql56-community

We verify and validate the upgrade process with various checks like monitoring information_schema and mysql database tables, installing plugins, verifying already installed plugins, defined users, running mysql_secure_installation, doing storage engine and version checks etc.

Upgrade from “old style” MySQL rpms to repo rpms with different MySQL versions

As we mentioned above, the repository rpms represent a deep rearchitecting of our “old style” rpms. We continue to support the old style packages, but we also give users a choice to “upgrade” to the repo style ones and thus simplify future upgrades and administration. To make sure this works seamlessly, we test all relevant version combinations of old style rpms and yum repo rpms (note: for EL7 there are no legacy old style packages, so this scenario does not apply there)

Upgrade from older to new MySQL packages in the MySQL repo

It is critically important that we don’t break the chain of upgradability between what we have delivered already and what we are delivering. This means that we need to test upgrades from older MySQL versions in the repo to new versions. For MySQL 5.6.20 and 5.5.39, this means:

mysql55-community(e.g. 5.5.38) → mysql55-community(e.g. 5.5.39)
mysql55-community(e.g. 5.5.38) → mysql56-community(e.g. 5.6.20)
mysql56-community(e.g. 5.6.19) → mysql56-community(e.g. 5.6.20)

(The precise combinations above will of course vary with the actual server versions in the repository at any given time)

Upgrade between 32 bit and 64 bit packages

We offer multilib support in our repo packages, meaning that users can install 32 bit and 64 bit packages side by side on a 64 bit system. In this testing we cover installation of individual 32 bit packages and verify that those still exist on the system after installing all 64 bit packages.

Third party application dependency testing

Most distros come with many third party applications which internally depend on some MySQL-like client, libraries or server being available on the system. Our testing in this area aims to ensure that dependent applications continue to work correctly after a user installs the real MySQL.

A good test strategy often involves reducing the number of combinations to be tested to a workable set, so we have been identifying four different dependency patterns and one  or more third party MySQL-dependent applications for each of these. We then check that these applications are unaffected by the installation.

These are the dependency patterns and their representative applications:

  • applications that depend on client and libs: dpm-name-server-mysql
  • applications that depend on libs: mysql-connector-odbc,mysql++
  • applications that depend on devel and libs: mysql++-devel
  • applications that depend on server, client, libs: ocsinventory

Overview of test scenarios and combinations

Here is the detailed test specification matrix for 5.6.20 repository package testing:

Type
Scenario
Test Scripts
Installnew server packages within repository like 5.5/5.6/5.7OracleMySQL_only
5.6.20_32bit_5.6.20_64bit_EL7
5.6.20_64bit_5.6.20_32bit_EL7
UpgradeUpgrade from native to current repository with different server version packages. Upgrade from native mariadb to 5.5.39. Upgrade from native mariadb to 5.6.20MariaDB_OracleMySQL
Upgrade between the server packages within same repository5.5.39_5.6.20-mysql-mysql
Upgrade from server packages in public repository to current repository5.5.39_5.6.20-public_mysql-mysql
Upgrade from regular rpms to current repository with different Server version packagesNo EL7 regular rpms available
Upgrade between 32 bit<-->64 bit packages within same server version or installing oracle mysql packages after every individual package Installation from the same repository of same server version but different architecture5.6.20_32bit_5.6.20_64bit_EL7
5.6.20_64bit_5.6.20_32bit_EL7
Upgrade between 32-bit<-->64-bit packages wrt native packages and oracle mysql packages update or installing oracle mysql packages after every individual package installation from native repo (mariadb) with 32-bit<--64-bit variations5.5.35maria_32_5.6.20_32_EL7
5.5.35maria_32_5.6.20_64_EL7
5.5.35maria_64_5.6.20_32_EL7
5.5.35maria_64_5.6.20_64_EL7
3rd party depApplications depending on:
client and libs-->dpm-name-server-mysql
libs-->mysql-connector-odbc, mysql++
devel and libs-->mysql++-devel
server, client, libs-->ocsinventory
the combination of the above 3-->ocsinventory
Mysql_56-thirdparty_EL7:
5.6_dpm-name-server-mysql
5.6_lfc-server-mysql
5.6_mysql-compat-server
5.6_mysql-connector-odbc
5.6_ocsinventory
Thirdparty-mysql_55_EL7:
Dpm-name-server-mysql
Lfc-server-mysql
Mysql-compat-server
Mysql-connector-odbc
Ocsinventory

So, what is it good for?

For an activity such as this, a key metric will always be cost versus benefit. The QA regime we describe in this posting is resource consuming, both in terms of initial investment and ongoing test development and execution for each release, and careful management of test scope and intense automation is important in order to keep especially the recurring execution costs down.

So, has it been worth it so far? The answer is a resounding “yes”, and we have data points to back that up. First and foremost, in spite of well over half a million downloads and a steady stream of new releases, the number of community reported issues has been very low. Of course, that doesn’t mean that we haven’t seen any issues, just that we tend to catch them before release. Here are a few bugs encountered during the 5.6.20 / 5.5.39 test cycle. Please note that this was the first test cycle run on the final GA version of EL7, so some issues were indeed expected:

  • MySQL bug #73073: update to community-server fails if mariadb-galera-server already installed
  • MySQL bug #73113: yum update fails when community repo enabled and mariadb-embedded-devel installed
  • MySQL bug #73194: 5.5 packages failed to install after installing mysql-bench from native repo

What’s next?

As the distro landscape develops and changes, our packages and our testing regime need to change as well. An example is the evolution of EL7: Red Hat made a number of big MySQL related packaging changes beginning with the EL7 beta version they released back in December 2013, to the release candidate in April and finally to the GA that appeared some weeks ago. For each of these milestones, we adjusted and expanded our testing in order to make sure that our packages worked as they should.

Going forward, we will of course continue to adapt to any changes. We will also continue to pursue efficiency by investing in more automation. In particular, we will add more automatic validation points at different stages of the install and upgrade process, we will enhance the system for test triggering and we will hook everything into our automated test reporting system. This should all hopefully allow us to spend even more time to ensure that we follow the distros closely and in general that we test the right things as thoroughly as possible.

And finally, we’d be delighted to hear your comments, questions and thoughts on all this. Ping us in the comments section!

Authors

This article was jointly authored by Ramana Yeruva and Yngve Svendsen from the MySQL Engineering team. We would also like to thank the MySQL Repository Project team for helpful suggestions and for valuable QA of the article contents.

They’re Here: Official MySQL Repos for Debian and Ubuntu

As we’ve hinted at for some time now, we have been busy preparing some good stuff for our Debian and Ubuntu users. And today we’re delighted to launch our own official MySQL apt repos for Debian and Ubuntu.

After working closely with the Debian and Ubuntu communities to make sure that the native MySQL packages in those distros work really well, we have now taken all the valuable knowledge and experience gained from that work and applied it to our own official packages.

We have support for Debian 7 and Ubuntu 12.04 and 14.04, and we initially offer MySQL Server 5.6 plus the latest 5.7 development milestone release. 5.6 is what we recommend for production use, while milestone releases are for those who want a taste of the absolutely latest. We will be adding more products in the near future.

So why are we doing this?

First and foremost, we want to provide officially supported, very high quality MySQL packages that integrate well with the distros, from easy-to-use repos that will always contain the latest and greatest; while distros typically choose one specific version of MySQL to include, we can offer several different versions, including development releases, from our repos. We think this is a nice complement to the good MySQL stuff that ships natively with the distros.

Second, running our own repos keeps us better hooked into what the community, and especially the different Linux distros, want and need from us. As part of our apt repo work, we’ve seen how we need to improve important stuff such as our startup scripts and config file handling in order to make life easier both for ourselves and for the Linux distro package maintainers. Going forward, we will continue to listen closely to what our users and our friends in the distros tell us, so that whether or not you use the native distro packages or our official repos, you will have a good and smooth experience.

What took us so long?

Well, learning takes time, and we wanted to begin by giving the distros full attention before we started work on our own stuff. And this is indeed complex stuff, with a huge matrix of possible migration paths and upgrades from distro packages, other vendors’ packages and our own old debs, all in multiple possible configurations, versions and editions.

When we put the “official” stamp on something it means that we have put a lot of time and effort into design and implementation and not least testing in order to make sure that things work as expected and that users have as smooth an experience as possible (there is a good blog posting on some of this over on the MySQL Server Team blog). We’ll hopefully at some point have a blog posting telling the story of how we translated that huge matrix into actual testing, but take my word for it: QA has been burning the midnight oil on this project.

Finally, while starting to use our repo will be very straightforward for probably 90% of our users, we have put together in-depth documentation that covers those few cases where the move requires a bit more care.

Want to get started?

Head over to the APT repo download page to get the setup package, install instructions and pointers to more in-depth docs.

There are a few minor limitations in these initial packages, where a tiny handful of distro native applications are incompatible with our repo packages. The docs have more details on this. We believe that this will affect very few users, but we will look at fixes for later revisions.

What’s next?

Over time, we will add more products to our repos and we will continue to improve the user experience. We will also continue to work closely with the Debian and Ubuntu maintainers and communities to make sure that we are doing the right things and that the native distro packages get even better in the future.

And finally, nothing in this world is perfect: if you have comments or suggestions for improvements, please let us know in a comment here, or if you run into issues using our packages and repos, please take the time to submit a bug report.

Congratulations, Ubuntu!

Today, we congratulate our friends at Ubuntu on a great new release, Ubuntu 14.04 LTS. As you can see in Mark Shuttleworth’s posting on Google+ from a few weeks back, MySQL has been cooperating closely with the Debian and Ubuntu communities to make sure that MySQL works very well on these platforms, and Ubuntu 14.04 bears the first fruits of those efforts.

Out of the gates, Ubuntu 14.04 comes with MySQL 5.5.35 as default, while 5.6.16 is available in the Ubuntu Universe repository. So whatever your MySQL preference – the mature and battle hardened 5.5 or the newer, richer and better performing 5.6 – Ubuntu 14.04 has what you need.

Going forward, Ubuntu is set to offer a great MySQL experience. Traditionally, Linux distros have tended to stay with a specific version of key components such as MySQL for a long time in order to avoid hitting possible issues introduced in newer versions. However, Ubuntu has recently introduced an approval process that will allow “trusted” software to get updated more often, and MySQL is now on that list of trusted products. Basically, Ubuntu has reviewed our development and QA processes and policies and concluded that upgrading to new versions of MySQL can be assumed to be safe. For Ubuntu users, this means that the distro repos should always have a recent version of MySQL available.

With 14.04 out and about, we really look forward to continuing our fruitful collaboration with the Debian and Ubuntu communities. We want to get MySQL 5.6 into Debian stable soon, and we think it would be natural to roll over to 5.6 as the default MySQL in Ubuntu 14.10. And we have some other good stuff in store for Debian and Ubuntu users pretty soon…

Repos and Distros: Upstream and Downstream

When we launched the official MySQL repos back in October, we wanted to achieve a number of things:

First, we closed a gaping hole in our distribution on Linux. Some Linux distros do not ship all the MySQL products, and not everyone is able to or always wants to use the distro packages. And the distros naturally focus on stable releases, while some users want to test development releases from us. Second, this represented a good opportunity to get off our behinds and finally modernize our Linux packaging. And last, but definitely not least, we would put ourselves in the same position as those valuable volunteers that take our products and integrate and maintain them in the Linux distros.

When you’re upstream, you can easily lapse into a mode where you stop listening properly to those who sit downstream and have to process what you’re releasing. There will always be some amount of difference in priorities and opinions between upstream and downstream, but putting ourselves downstream and in that sense having to “eat our own dogfood” turned out to be very educational and highly useful.

This change in perspective made it easier for us to understand many of the important pain points of the distros. It quickly became clear to us that a good deal of those pain points could be fairly easily addressed, and in some of the more difficult cases, the folks involved in the repo project have acted as lobbyists internally in order to have Engineering priorities changed so we could fix some of the bigger things.

Below, I will go into some more detail on what we are doing and what we have accomplished in the Debian and Ubuntu communities, let me first just say that we hope that we are being considered useful citizens in the different distro communities, and that distro maintainers of MySQL products see us an approachable and responsive upstream. And if we aren’t, let us know!

Debian and Ubuntu

Over the last couple of weeks, we have seen our new experience and efforts bearing fruit in Debian and Ubuntu. Since late fall last year, release engineers and developers from the MySQL team have been working with the Debian and Ubuntu community to bring MySQL 5.6 into both distros. Two weeks ago, Ubuntu bug #1256931 (“please include MySQL 5.6″) got closed, and as you can see from the package status, 5.6 is now headed for Ubuntu 14.04 LTS, Trusty Tahr.

Hot on the heels of the Ubuntu news came the inclusion of 5.6 in Debian experimental. We do believe that the current packages are quite solid, and we will continue to work closely with the Debian community to deal with any reported issues and in general refine things so that 5.6 can go into Debian testing soon.

Working with the Debian and Ubuntu folks has been an absolute pleasure, and at the risk of leaving out someone, I will say a heartfelt thanks to Robie Basak, Björn Boschman, Clint Byrum, Dustin Kirkland and James Page, and to our very own Akhil Mohan and Norvald Ryeng for their hard work.

Maintenance and Updates

And now that we have 5.6 in Ubuntu, how will it get maintained there? MySQL maintenance releases come out on a bi-monthly schedule, containing fixes for important bugs reported by the community, by customers and by our own developers. This is not a perfect match for Linux distros, where the approach has historically been to select a specific MySQL release to go into a new major release of the distro. Subsequent maintenance has largely consisted of the distro maintainer fixing reported issues or security bugs by either backporting from later MySQL releases or implementing their own fixes.

There are many good reasons for this approach, and a key one is trust. If a distro maintainer were to update the distro packages more or less in sync with upstream releases (for MySQL, “upstream” is us), he or she would have to be able to trust that upstream doesn’t invite trouble by introducing risky bug fixes or new features in maintenance releases, that upgrades and forward compatibility don’t break, and that upstream in general has development and QA processes in place to prevent any surprises in new releases.

We are happy to say that we have earned that trust in Ubuntu. Earlier this month, MySQL was granted a so-called “micro release exception”, which means that new MySQL maintenance releases should appear on a regular basis in Ubuntu going forward. The relevant mail thread is on the Ubuntu tech board mailing list, and you can read much more about MySQL’s maintenance release process in an excellent post over on the MySQL Server Team blog.

What’s Next?

Right now, we’re taking all the experience and knowledge we have gathered while working with and in the Debian and Ubuntu community and applying that to our own debs. The current debs were in much need of some love, and we’ll have some very good news for our Debian and Ubuntu users in a little while.

In parallel with that, we are continuing the work to further refine the current repo packages, add more platforms and more products to the official MySQL repos, and in the process also try to reach out to distro maintainers of the same products to make sure we understand what life downstream is like. And we will continue to work with distros in areas beyond packaging.

Requests, ideas, thoughts on our packaging efforts? Anything else we should be doing together with the distros? Let us know in a comment!

Utilities! Fresh Utilities! Get Your Utilities Here!

MySQL Utilities is one of those gems that every MySQL user should know about. They simplify and help automate a wide range of tasks and activities that everyone from casual users to seasoned DBAs may need to carry out on their MySQL setup from time to time, all the way from cloning users, copying databases and exporting data to managing replication topologies and failover.

This has been on our list of products to put into the MySQL repos for a long time. But the collection of utilities  is evolving quickly, with new features and improvements being added all the time, and when I’ve approached the dev team to get the repo effort underway, they’ve always been responding “Hey, can’t we wait until the next release? We have all these cool new features lined up for that!”

Well, we finally convinced them that what they have now is more than good enough to go into the repo. So as of this morning, you can go to the repo setup page and grab the relevant setup package, and within minutes, you will have at your disposal some pretty powerful stuff to help you maintain and run your MySQL system.

Want to know more about MySQL Utilities? Here are a couple of pointers:

And as usual, drop me a comment if you have viewpoints on the MySQL repos in general or the Utilities packages specifically. And should you (*shudder*) encounter problems when using our repos, please submit a bug report.

Giving Red Hat 7 Users a Choice

Red Hat recently released a beta of what will become the next version of their Enterprise Linux. As many people have noticed, Red Hat opted for a forked version of MySQL 5.5 (which GA’d 3 years ago).

MySQL has come a long way in those three years, and MySQL 5.6 was released just under a year ago. It has much improved performance, it is easier to manage and it has a boatload of new and enhanced features across the board. And not least: over this last year it has proved itself to be the most stable and robust MySQL release ever.

The problem is that Red Hat 7 beta users aren’t given a choice: There is no real, recent MySQL Server in the distro. We certainly welcome competition in the MySQL area; healthy competition energizes people and drives them to perform, but that requires giving users a choice. Restricting choice restricts competition and hurts users.

So today we are launching support for the RHEL 7 beta from the official MySQL yum repository, giving users the choice between what comes with the distro and a more modern, robust and well performing product: MySQL Server 5.6. We also added MySQL Server 5.5 to provide a smooth upgrade path and give a choice to those who want to run an older version of the real MySQL. And to top it off, we have the latest Server 5.7 development milestone in there as well, for those who want to get a taste of the very latest and greatest.

With MySQL Server plus the ODBC and Python connectors in place for RHEL 7, we will proceed to add more connectors and tools over time. So keep watching this space for more news.

Now, if you are an early adopter of EL7, head over to the repo setup page for an easy way to get the real MySQL on your system. And please leave your comments here, or submit a feature request or bug report to the MySQL bug database.

Fedora 20 Is Here: We Support It From Day One

Today, we congratulate the Fedora community on their newest release. We believe it is a great platform to run the real, official MySQL on, and today we are adding Fedora 20 support in our Yum repos.

We have the latest MySQL Server 5.6, the latest 5.7 development release and Connector ODBC ready for you on Fedora 20. Workbench is coming, it just needs a little bit of tweaking first.

In addition, we have added Connector/Python to the repo for all supported platforms.

Now, head on over to the repo setup page on dev.mysql.com to get the real stuff!

MySQL Repo News

What’s new in the MySQL Yum repositories

A little over a month ago we launched the official MySQL Yum repository. An official repo was a long standing request from the community, and we’ve had it on our list of todos for a long time. We finally got it done, and in the process we modernized and improved our RPMs vastly and hopefully also made life a bit easier for the long suffering distro package maintainers, volunteers that provide the all important service of keeping MySQL fresh and usable in the Linux distros.

The greatest challenge in the project was to get to initial launch. Of course, a stable and reliable service is very important, but even more essential was that we wanted to deliver packages that really represented a leap forward in usability, integration with the underlying system and that not least closely matched the expectations of Linux users anno 2013.

We believe that we got at least part of the way towards those goals; uptake has been good and there are now thousands of systems out there running on packages from the MySQL repo. We have received valuable feedback, both negative and positive, and we have hopefully managed to respond to questions and help resolve problems in a good way so far.

Last week, we took the next major steps in the repo project, largely based on feedback and requests from the community. Here is a list of the major changes and additions:

Server milestone releases in the repo

  • We are now releasing MySQL Server development milestones through the repo: the just released MySQL 5.7.3 is available now
  • Simply enable in your repo setup. When the next milestone comes out, simply do a yum upgrade
  • Milestone RPMs in the repo and RPMs downloadable from dev.mysql.com are now identical

More platforms and easier upgrades

  • We added MySQL 5.6 packages for EL5. So the repo now has MySQL 5.6 on Red Hat Enterprise Linux 5, Oracle Linux 5, CentOS 5 and on Fedora 18 and 19 (stay tuned for Fedora 20)
  • We added MySQL 5.5 packages for EL6. As EL6 ships with MySQL 5.1 “out of the box”, this provides a smooth upgrade path to our users. Edit your repo configuration and do a yum upgrade. One or more repeats will take you to whatever point you want in the sequence 5.1 > 5.5 > 5.6 > 5.7.

Better user control of product versions

  • We split the repo by MySQL Server version. You set the major version explicitly in the repo config on your system – no one wants an accidental run of yum upgrade to perform a major version upgrade unexpectedly. MySQL 5.6 is the default.
  • Simple workflow to move through version upgrades (see above)
  • We split out separate Tools and Connectors repositories

More details

Please let us know about your experiences with these new repositories, as well as any suggestions you may have to improve the experience. Add your comments here, or to report specific bugs or problems or request features, use bugs.mysql.com (there is a new “MySQL Repositories” category).