This was written in response to Michael "Monty" Widenius' post at
http://www.devshed.com/BrainDump/MySQL_Benchmarks/, where he attempts to
contest the Great Bridge, LLC's benchmarks of RDBMS'es.

> == Monty

> I would really like to comment against the last PostgreSQL benchmark
> test, if there would be anything to comment. As they haven't released
> any information about the test or even the results one can just assume
> that they have done a test that doesn't have anything to do with the
> real world :(

About the Great Bridge Benchmarks:
    http://www.greatbridge.com/news/p_081620001.html

> All database has some good and weak areas and it's very simple to
> design a test that test the things other are weak at. MySQL 3.22 is
> for example not very good for mixed inserts + selects on the same
> table and that is also the only thing we ever have seen PostgreSQL
> users test in their previous tests when PostgreSQL comes outs as the
> better. PostgreSQL has of course also a lot of weak areas, like slow

From http://www.postgresql.org/mhonarc/pgsql-hackers/2000-08/msg00331.html:

    The AS3AP test is a *read only test*, which should have been MySQL's
    bread and butter according to their marketing literature. The shape of
    that curve shows that MySQL started wheezing at about 4 users, and
    tailed off rapidly after that point. The other guys barely made it out
    of the starting gate :)

> connection, slow insert, slow create/drop tables, running long
> multiple transactions where you get a conflict at the end (in this
> page/row locking is better) and the need to run vacuum() from time to
> time (especially when you do a lot of deletes / updates). We will
> update our benchmark comparison web page with PostgreSQL 7.0.2
> shortly, to give another picture of this story. The only way to know
> which database is best for ones application is to write a simulation
> of the applications and test both databases.

> We here at MySQL has always tried to design very fair test that no one
> can misinterpret or lie about. Anyone can download our benchmarks and
> run themselves to verify our test results. One could argue that we

It would be preferable if the banchmarks that come with the MySQL
distribution would actually _run_ against other databases. I spent hours
trying to get the perl scripts to run against anything other than MySQL,
with no luck whatsoever.

> don't test everything, but we do add new tests all the time and we
> also accept test from database users that wants to test some specific
> things against a lot of databases. This will in the long run make our
> tests very fair and easier to use as a decision tool when choosing a
> database.

> It's a shame that Great Bridge funds a test that is solely done to
> confuse users instead of telling the truth; PostgreSQL is good in some
> areas, and bad in others, just like all other databases. No database
> can be used to solve all problems (at list not without providing a lot
> of hints for the database and a lot of dedicated code to solve the
> problem) and you can't win all test. If one starts claiming this one
> is on very thin ice!

I think the folks at Great Bridge have been very open about their test
results, and other information about the test. They have several times
come forth with specific information about their tests. I don't think that
they are trying to hide anything, nor trying to make the benchmarks biased
one way or another.

> I hope that Great Bridge doesn't get a lawsuit from the TPC
> organization by misusing their test. This may backfire on all open
> source databases as I have never seen any commercial database does
> such a bad press release and the commercial companies may think that
> this is the way to do it in the open source world :(

From http://www.tpc.org/faq_general.html:

    Q: If I want to run a TPC benchmark, do I have to join the TPC?

    A: No. TPC benchmarks are industry standards. The TPC, at no charge,
    distributes its benchmark specifications to the public. Anyone is free
    to implement and publicize a TPC result.

> The way to set up the databases in a test is also very crucial for the
> performance of the database. The article doesn't mention anything
> about this or even with which ODBC driver they used the different
> databases. The defaults for MySQL is for a database with moderate load
> and it should take very little resources. MySQL also have two ODBC
> drivers, one slow with debugging and one fast. It should be very nice

From http://www.postgresql.org/mhonarc/pgsql-general/2000-08/msg00261.html:

 ... Marc's right... we opted for ODBC to ensure as much of an "apples to
 apples" comparison as possible.  Of the 5 databases we tested, a native
 driver existed for only the two (ahem) unnamed proprietary products -
 Postgres, Interbase, and MySQL had to rely on ODBC.  So we used the
 vendor's own ODBC for each of the other two cases...

> to know how they actually did use MySQL. To get any performance from
> Oracle, on has also to tune this a lot; The ODBC driver for Oracle has
> also very bad performance; This is a common known fact; No one runs a
> critical system with Oracle and ODBC. (I use Oracle here as an
> example, as they press release implicate that they are using Oracle
> for testing; If not, they are not testing against the proprietary
> database leaders).

> One thing that also is interesting is that they don't mention which
> PostgreSQL version they are using. It's very unlikely that they did
> actually test PostgreSQL 7.0 as this has at least one very fatal bug
> in the index handling which made it useless for benchmarks (at least
> when we did a test run on it). If they have tested another version, a

From http://www.greatbridge.com/news/p_081620001.html:

    What versions of the databases did you use? 

    PostgreSQL - 7.0 release version
    Proprietary 1 - 8.1.5
    Proprietary 2 - 7.0
    MySQL - 3.22.32
    Interbase - 6.0

> tuned PostgreSQL with non standard patches and non standard setup,
> they have broken even more rules. According to the give benchmarks
> numbers, it looks as if they have run PostgreSQL without disk syncing,

From http://www.postgresql.org/mhonarc/pgsql-general/2000-08/msg00289.html:

    Tatsuo Ishii wrote:

    > BTW, was the postmaster configured to have an option "-o -F" to
    > disable fsync()?

    Good question, Tatsuo... We ran it with and without fsync() - there
    was only a 2-3% difference between the two.

> which would be fine if they would have told it in the article and if
> they would also have done this with the other databases.
> I agree that MySQL 3.22 is weak in the case where you do a lot of
> inserts + updates + selects one table, but it's on the other hand

See my quote earlier. The AS3AP test is SELECT (i.e. READ) only. Even if
they had disabled fsync(), what impact would this have on a disk READ...

> equal strong if you don't do that; In MySQL 3.23 you can intermix
> insert + select but not updates. This will be fixed with BDB tables,
> but a release with this is still a few weeks into the future. Threads
> also gives MySQL better scalability than processes, that PostgreSQL
> uses, so we are very confident about the future.

> I also don't agree with the argument that they are not testing MySQL
> 3.23 as this is still marked 'beta'. We have here at MySQL a
> completely different release schedule on our versions. We don't mark

From http://www.mysql.com/downloads/mysql-3.23.html:

    Downloads for the 3.23 version
    -----------------------------------------------------------------------
    Note that this version is still in heavy development and may have bugs
    in the new features introduced in this version. It may also change in
    incompatible ways between versions. It does however pass our standard
    test suite so it should not be completely broken :)

If you don't consider it beta, don't call it "beta."

> anything as release if there has been any significant bug report for
> the gamma version for a month. Compared to our release schedule,
> PostgreSQL 7.0 would be called alpha (I don't mean anything bad with
> this; Many users users our alpha version in a production environment
> with good results)

> The net result is that the posted test is about as wrong as you can do
> a test, the important thing is just to get the people that reads that
> page to understand that.

> Regards,
> Monty

Regards,
Dominic