Feed aggregator

DrizzleDB: Drizzle source tarball 2011-07-21 has been released http://t.co/8adtU9e

This last March, David Shrewsbury put together a basic implementation of multi-master replication into DrizzleDB link. We were able to actually merge it with the mainline trunk, but then had to remove it until we had refactored and fixed more things. Today, we were able to put it back in.

It's worth reading Shrew's original blog posts, and then trying it out.

Drizzle source tarball, version 2011.07.21 has been released.
In this release:

• Continued code refactoring (thanks again to Olaf van der Spek)
• Continued work on the stored procedure interface (yay Vijay!)
• Improvements to the Storage Engine API tester from Stewart
• Various bug fixes

The Drizzle download file can be found here

It’s notoriously hard to measure the usage of open-source software. Software that’s open-source or free can be redistributed far and wide, so the original creators have no idea how many times it’s installed, deployed, or distributed. As a proxy, we often use downloads, but that’s woefully inadequate.

I’ve recently begun trying to figure out how many job openings are mentioning various open-source projects. I think that this might be a better metric because it’s driven by the end result (usage), rather than intermediate processes (downloads, etc). I think that it’s likely that usage and demand for skilled people is somewhat realistically related.

To be more concrete, I’ve been watching RSS feeds from job posting aggregators for several alternative versions of MySQL: Percona Server, MariaDB, and Drizzle. It appears that Percona Server is by far the most in-demand in terms of job skills. (I haven’t seen a job posting for the others at all, so far.)

On the other hand, my sample is skewed; I think Percona Server is better known in America, but MariaDB might be more visible in Europe. And I’m not sure that the sample data set is large enough to be statistically significant. Percona Server jobs are utterly dwarfed by MySQL jobs.

There are other flaws in my method: some software doesn’t really need as much manpower to run as others. I would say that given an equal number of WordPress and Drupal websites, more of the Drupal websites are going to be trying to hire experts to manage their sites. So nothing is apples to apples.

What do you think about this metric and its merits or drawbacks? Is there a better way to figure out how much adoption a project really has?

Related posts:

1. Does MySQL really have an open-source business model?
2. What does an open source sales model look like?
3. Open-source database developer mailing lists
4. Making Maatkit more Open Source one step at a time
5. MySQL: Free Software but not Open Source

An ideal summer day is when the sun is shining, the warm sunshine refreshes your body, the light breeze becomes naught with your hair, the birds sitting on intoxicated leaves smile at you, and you to top all of that you get the latest hot issue of the Log Buffer. Yes to put cherry on your cake, Log Buffer #227 is here.

Oracle:

David Kurts tells us about applying Hints to Objects inside Database Views.

Jonathan Lewis answers another burning question on on multi-column bitmap indexes and the inability of Oracle to create a bitmap index join when (to the human eye) the strategy was an obvious choice.

A blog post about the mapping of Physical Disk, LUNS, Cell Disks, and Grid Disks in Exadata by lovely Oracle.

Assume that you had a detail table that contained several attributes for each of the unique key values.  How would one go about finding all of the unique key values that share the same set of attributes? Charles Hooper answers.

Randolf Geist gives a short heads-up note regarding a bug that obviously has been introduced with 11.2.0.2.

SQL Server:

Mark Broadbent has posted a tribute to the blogger Jen Stirrup, how has just been recognized by Microsoft by their MVP Award program and become a Most Valuable Professional.

ROW_NUMBER() can be used to generate a sequential number for each row in the result set. Unlike RANK() and DENSE_RANK(), ROW_NUMBER() in case of ties it does not generate same number, it simply ignores the tie and generates sequential numbers for each of the tied rows. Vishal gives an example.

Jason Brimhall is giving a interesting spin in the realm of SQL statistics.

Juneau is the code name for the new SQL Server Development Tool (SSDT), to be released along with the next version of SQL Server, codenamed “Denali“. James Serra has an interest overview.

Wes Brown is looking at assembling a basic data dictionary from the column level meta data stored in SQL Server.

Pradeep Adiga sheds light on SavetoSQLServer method error in his blogpost with a story to tell.

MySQL:

Mark Atwood shares his valuable thoughts on node.js and libdrizzle.

Michael “Monty” Widenus blogs about his last MariaDB 5.3 feature before they go beta and that is Progress reporting for ALTER TABLE, LOAD DATA INFILE etc.

In order to optimally size the amount of RAM to allocate to a set of new machines for running MTR, Jonathan Perkin ran a few tests to check the memory usage of an MTR run for mysql-trunk and cluster-7.1.

By far, the most popular way for PDI users to load data into LucidDB is to use the PDI Streaming Loader. Nicholas Goodman has a good blog post about this.

Haidong Ji has presented a comparison of HandlerSocket and mysql client libraries with Python.

Happy Summer !!!

Today I was chatting with a technical friend, and she mentioned that she has a "todo" of writing a decent node.js driver for MySQL. A bit more chatting back and forth, and http://nodejsdb.org/db-drizzle/ was discovered. It was written to use libdrizzle and drizzled, but since libdrizzle can talk to MySQL, it should work for her needs as well.

I would love to see some work done on how well libmysql+mysqld, libdrizzle+mysqld, and libdrizzle+drizzled handle highly concurrent asynchronous event-oriented workloads such as those generated by all these new node.js applications.

I suspect that all sorts of surprising bugs will be discovered.

Please help us discover those bugs.

I have been running the test and merge process for DrizzleDB using Jenkins.

Jenkins is a pretty standard Java-based web app. The configuration settings are stored in XML files, and that configuration is manipulated using "easy to use" Web GUI.

The "old skool" UNIX-like way to keep configuration settings is in a text file, which is edited with an ordinary text editor, and is read by the program daemon on start or SIGHUP. This is considered "scary", "hard to learn" and "hard to use" by novices.

There is a big problem with GUI-only managed configuration, text file configuration has a major advantage.

I did not set up the Jenkins server or nodes. I am not the only person with admin access to it. Several other people have set it up, set up various projects in it, and added new nodes and new types of nodes.

As I work on it, and look at the existing configuration, I often find things that are "surprising", things that make me say "Is that right? That can't be right? Can it?". And then I have to spend time digging into it. Something it IS right, for reasons I didn't know at that moment. Sometimes it used to be right, but isn't necessary any more. And sometimes, it just wasn't right.

In a textual configuration file, you can put comments. The purpose of a comment is to communicate into the future, to tell those who came after you (including your future self) what you were intending to do, and why you selected some "surprising" option or way of doing things.

There is no good way to put comments into GUI or WebGUI configuration, even if it has a freeform field labelled "comments".

DrizzleDB: RT @Gazzang_Inc: A game changer @cassandra @mongodb @drizzledb @MySQL @planetpostgres @Alfresco @JBossNews @drupal http://ow.ly/5mXkg

DrizzleDB: RT @stewartsmith: MEDIUMINT - fixed in #drizzle (as in it's dead. It's passed on. It's ceased to be, expired and gone to meet its maker.

DrizzleDB: RT @star_d: enjoying #osb11 presentation on #Drizzle by @brianaker

Well, that’s pretty much it, thanks for stopping by ; )

In all seriousness, it’s kind of neat that we’re using dbqp to run some of our staging tests and we gain a few neat things:

Speed

Here are the trend charts for randgen and crashme.  While it doesn’t look like randgen is showing much of an improvement, it is worth mentioning that this job now runs both the standard and the transaction log tests in a single run >: )  Previously, we had a separate drizzle-automation job for the transaction log.  Just the trx_log tests took ~30 minutes to run (plus build time).  Long story short, we’re saving about 30-40 minutes on randgen testing per staging run and only needing to build once!

Maintainability

The jobs we run are in the tree and anyone can easily repeat them.  While Drizzle-automation kicks major butt (and I have taken many ideas from it), it is a separate piece of software that requires setup and maintenance.  Basing things around an in-tree setup means that you only need the code and any required bits and pieces.  Now if we need to set up a new randgen machine, we only need the randgen and dbd::drizzle installed (and we plan on including randgen in-tree soon, so you won’t even need that!).  If we need to set up a new crash-me machine, we only need dbd::drizzle – and everyone should have dbd::drizzle installed! ; )

Ease of use

Pretty much all tests provide the same standard output:

dtr mode

From the command:

./dbqp

Our default mode is dtr (aka using drizzletest.cc to execute standard .test files). To run all available tests, use the make target – make test-dbqp

20110621-081404  trigger_dictionary.loaded                  [ pass ]       43
20110621-081408  logging_stats.cumulative                   [ pass ]     1045
20110621-081412  errmsg_stderr.stderr                       [ pass ]       36
20110621-081412  ===============================================================
20110621-081412 INFO Test execution complete in 496 seconds
20110621-081412 INFO Summary report:
20110621-081412 INFO Executed 566/566 test cases, 100.00 percent
20110621-081412 INFO STATUS: PASS, 566/566 test cases, 100.00 percent executed
20110621-081412 INFO Spent 254 / 496 seconds on: TEST(s)
20110621-081412 INFO Test execution complete
20110621-081412 INFO Stopping all running servers...

randgen mode

From the command:

./dbqp --mode=randgen --randgen-path=/path/to/your/randgen


20110621-170141  main.subquery                              [ pass ]     3780
20110621-170148  main.subquery_semijoin                     [ pass ]     3016
20110621-170156  main.subquery_semijoin_nested              [ pass ]     3750
20110621-170202  main.varchar                               [ pass ]     2658
20110621-170202  ===============================================================
20110621-170202 INFO Test execution complete in 147 seconds
20110621-170202 INFO Summary report:
20110621-170202 INFO Executed 19/19 test cases, 100.00 percent
20110621-170202 INFO STATUS: PASS, 19/19 test cases, 100.00 percent executed
20110621-170202 INFO Spent 77 / 147 seconds on: TEST(s)
20110621-170202 INFO Test execution complete
20110621-170202 INFO Stopping all running servers...

crashme mode

From the command:

./dbqp --mode=crashme


20110621-181515  main.crashme                               [ fail ]   149840
20110621-181515  func_extra_to_days=error        # Function TO_DAYS
20110621-181515  ###
20110621-181515  ###<select to_days('1996-01-01') from crash_me_d
20110621-181515  ###>2450084
20110621-181515  ###We expected '729024' but got '2450084'
20110621-181515  func_odbc_timestampadd=error        # Function TIMESTAMPADD
20110621-181515  ###
20110621-181515  ###<select timestampadd(SQL_TSI_SECOND,1,'1997-01-01 00:00:00')
20110621-181515  ###>1997-01-01 00:00:01.000000
20110621-181515  ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
20110621-181515  ###
20110621-181515  ###<select {fn timestampadd(SQL_TSI_SECOND,1,{ts '1997-01-01 00:00:00'}) }
20110621-181515  ###>1997-01-01 00:00:01.000000
20110621-181515  ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
20110621-181515
20110621-181515 ERROR Failed test.  Use --force to execute beyond the first test failure
20110621-181515  ===============================================================
20110621-181515 INFO Test execution complete in 153 seconds
20110621-181515 INFO Summary report:
20110621-181515 INFO Executed 1/1 test cases, 100.00 percent
20110621-181515 INFO STATUS: FAIL, 1/1 test cases, 100.00 percent executed
20110621-181515 INFO FAIL tests: main.crashme
20110621-181515 INFO Spent 149 / 153 seconds on: TEST(s)
20110621-181515 INFO Test execution complete
20110621-181515 INFO Stopping all running servers...

While this isn’t a huge feature, it is nice to have a standardized report for knowing if something failed, what failed and how (we always dump test tool output on test failures).  Why is this nice?  Well, the world is a busy place and only needing to know one way of reading test output simplifies things just a teensy little bit.  This small improvement becomes a huge benefit over time if you happen to spend good chunks of your day looking at test output like me : )

Other than that, I’m still working on teaching dbqp interesting new tricks that will help me in testing SkySQL‘s Reference Architecture – expect to hear more about that next month!

Drizzle source tarball, version 2011.06.20 has been released.

This is a Fremont development release.  Our stable GA release can be found here.

In this release:

• NOTE:  Removed the drizzleadmin utility has been removed.  The same functionality can be achieved via cleaner, alternate methods (UDS and the console plugin + ssh)
• DATA_DICTIONARY tables' identifiers now use MAXIMUM_IDENTIFIER_LENGTH to match INFORMATION_SCHEMA behavior
• Continued code refactoring - thanks to Olaf van der Spek for his contributions here
• Various bug fixes
• The dbqp test runner now has sql-bench and crashme modes.  Docs are here.  You can read more here.

The Drizzle download file can be found here.

DrizzleDB: Drizzle source tarball 2011.06.19 has been released. http://bit.ly/lfIaDc

DrizzleDB: @StewartSmith newly of Percona, and one of the core #drizzle developers will speak at #OSCon. http://bit.ly/lejShZ, http://bit.ly/lNpfH1

DrizzleDB: Less than one week! Talks by Data Differential CTO, Brian Aker on #drizzle and #gearman at #OSBridge June 21-24. http://bit.ly/mxXdER

DrizzleDB: Drizzle testing - now with more server stressing goodness! http://bit.ly/jhqruG

DrizzleDB: http://bit.ly/myeQQa It turns out, that people might actually care about making sure that the data they thought was saved, was saved.

In December I covered the topic The state of MySQL forks: co-operating without co-operating (which was also a response to Giuseppe Maxia's take on the same topic). Since half a year has now passed, I was wondering if I should follow up with an update. (Drizzle having a GA release would be the major news in such an update.)

But I see that Peter Zaitsev covered this topic in the opening keynote of their Percona Live conference. Since I agree with Peter's view on this topic, I just recommend you watch the talk on Percona.TV. He also uses the same categorizations of the forks, and includes "community patches" as its separate slide.

read more

DrizzleDB: #Drizzle will have a booth at #OSCon Data July 25th - 29th at the Portland Convention Center. Come meet the Developers!

DrizzleDB: RT @DDifferential: http://bit.ly/jhqruG More Drizzle testing, with stressy goodness!