Select Page

MySQL European Customer Conferences 2007

Today I attended the MySQL European Customer Conference in London. I was quite excited about it hoping for some good tech sessions and such, especially after seeing the lineup:

  • MySQL roadmap
  • MySQL performance tuning including real world examples
  • MySQL for online applications
  • MySQL for data warehousing and bi
  • Presentation by the Swedish Police
  • Presentation by Net-A-Porter DBA
  • High Availability Strategies for MySQL
  • Multiplay Telecom Service Architectures with MySQL Cluster Carrier Grade
  • Defining an Effective Storage Engine Strategy

Thats a pretty decent line-up, seemed like a lot of solid tech info in there. I chose to attend the MySQL Roadmap, MySQL for online applications, Net-A-Porter, Multiplay Telecom Service Architectures with MySQL Cluster Carrier Grade and Defining an Effective Storage Engine Strategy
The sessions by MySQL employees were good, obviously done by senior sales type guys but all knowledgeable. The sessions were though aimed at tech management types rather than pure tech people, they contained a lot of what can only really be described as sales talk. This held true for the Datawarehousing, Storage Engine and obviously the roadmap talks.
The session on MySQL for online applications could have just been left out really, nothing was really said of any substance, the basic building blocks of a web architecture was mentioned but just not in any sufficient detail.
I found the data warehousing session good as I am currently running into some needs for this kind of application, the roadmap was very helpful if only to show me that some of the much needed improvements is way down the line ๐Ÿ™
I attended the Pret-A-Porter talk as I was hoping to get some insight into the workings of a big MySQL shop. Turns out the DBA who was running the chat was new in the job and can’t really comment on any of the decisions that got them where they are today. He came up with such gems as ‘to me Hibernate is just this horrible little thing the developers use’ and couldn’t really describe why he made some choices like use InnoDB for everything. When asked how he optimised the database he just said he followed the basic recommendation the MySQL Enterprise Monitor suggested, couldn’t really go into any details. In fact he didn’t even know what hardware they run on.
Further I question the choice of partner for this specific presentation, these guys have about 2GB of data and it all fits into the query caches, they have one database server and is considering moving to a DRDB HA cluster, again he can’t really say why thats just what he was told to do. Frankly, running a MySQL server with 2GB data that performs well is no challenge and few people in the industry that actually use MySQL can learn from this presentation.
I chose the Telecoms Strategies talk as it was touching on MySQL Cluster which is something I have read about but never had opportunity to use. The presentation was well rounded, delivered professionally and gave me a lot of insight in the underlying NDB storage engine that enabled MySQL Cluster. The guys focus on talking direct to the NDB using the API rather than SQL so not quite typical but it was really interesting to get a view on the structure of the engine and how it sticks together. It made it crystal clear to me that Cluster would be a engine you would only use in very specific work loads. This was confirmed later on in the conf.
Overall, I feel I didn’t get value for my buck from the conference. Very little that was said had any bearing on my database that is approaching 100GB with 5000 queries per second.
A co-worker attended the performance sessions, he left equally unimpressed. The guys from MySQL Performancing Blog were constantly having to correct the speaker on his points.
So, again, pretty unimpressed. I think I’d need to attend one of the User Conferences which should have a more tech focus at least in parts ๐Ÿ™

Exim on CentOS 4

I recently bought a new machine from Layeredtech for my commercial mail anti spam system and am having endless troubles with it. I have a similar machine at Hetzner also running CentOS 5 and it too is having problems, though less frequently.

The short of it is that the drives disconnect, file systems go read only and the box needs a reboot:

ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
ata1.00: (BMDMA stat 0x4)
ata1.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 (timeout)
ata1: soft resetting port
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: qc timeout (cmd 0xec)
ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
ata1.00: revalidation failed (errno=-5)
sd 0:0:0:0: SCSI error: return code = 0x00040000
end_request: I/O error, dev sda, sector 226813249
Buffer I/O error on device sda3, logical block 27835568
lost page write due to I/O error on sda3
sd 0:0:0:0: SCSI error: return code = 0x00040000
end_request: I/O error, dev sda, sector 227360961
Buffer I/O error on device sda3, logical block 27904032
lost page write due to I/O error on sda3

So in an effort to figure out if this is a CentOS 5 problem – both ISPs certify CentOS 4 on their hardware – I needed to get my application going on CentOS 4. This turned out to be quite a mission involving getting Exim with MySQL and the recently integrate exiscan rather than the patched version.

I looked at the various options and decided to just backport CentOS 5’s Exim package to CentOS 4.

As it turns out I haven’t yet had a machine re-installed with CentOS 4 as I found some posts suggesting some kernel parameters that might fix things, I’ve applied these now to the machines and wait.

My Exim RPMs can be found below:

exim-4.63-3.src.rpm
exim-4.63-3.i386.rpm
exim-mon-4.63-3.i386.rpm
exim-sa-4.63-3.i386.rpm

As with the CentOS 5 ones you’ll need various DB client libraries installed as this supports speaking to Postgres, MySQL, SQLite etc.

This should be useful to anyone who just wants a more recent version of Exim on their CentOS/RedHat 4 machines.

New Server and Site Location

This is just a quick heads-up that I am moving this site and most of my services that I offer people to a new server this means for a while things will be odd.
The new machine is still with Blackcat Networks who I never hesitate to recommend to anyone. The new machine was purchased from DNUK who specialize in Linux servers, their service were great and managed to build me a machine that would fit into 0.5Amp power usage easily.
The final spec is a AMD X2 3800+, 2 x 250GB drives and 2GB memory, all happily running at around 0.4Amp. The old server is a Dell 1550 1Ghz PIII, so this will be a very welcome upgrade.

Google Apps for Domains Part III

This is part 3 of my series of posts on the general usability of Google Apps for Domains as a hosting service for email. This one really speaks for itself so I’ll keep it short.


It’s a bit like those joke Windows error messages that does the rounds.

Google Apps for Domains Part II

This is the 2nd in a series of posts about Google Apps for Domains, I’ve been evaluating it for a while. Today I’ll focus a bit on the XML feeds that Google Calendar outputs as a means of integrating it into other applications.

It basically provides 2 types of XML – a private and a public one. The public one only works if you share the cal publicly and the other is for your own use to pull into apps of your own. Each calendar should have a private one.

This sounds very interesting, so what’s the problem? Well there are some inconsistencies. On my two domains, the one has private URLs on the main calendar the other doesn’t have the option at all. So this means my primary calendar doesn’t have a XML feed unless I share it publicly.

GCal also has a feature that mails you a short daily agenda, much like what you see if you hit the Agenda button. Problem is the agenda mail function ONLY reads from your primary calendar, it does not include events from any subscribed calendars etc. This means that one of the biggest selling points of Google for Domains is crippled, if you share calendars you can’t use them even in the rudimentary tools provided.

I can’t use my primary private calendar with XML feeds, and I can only get events mailed from the primary calendar. Why is this a problem? I want to use the shared calendars with people I work with, that’s one of the major reasons to use Apps for Domains.

On my iCal calendars I had a RSS feed that I’d pull into some perl code to generate agendas. I resigned myself to the fact that I’d need to write one myself again now for GCal and wanted to use the XML feeds. I can’t pull events into my primary calendar – it doesn’t have private feeds – so I had to create a new cal and use that – completely invalidating the in UI agenda function. I also can’t delete the primary function, the option is grey’d out.

Back to the feeds, by default the data you’ll get in a feed looks like this for recurring events:

<?xml version='1.0' encoding='UTF-8'?>
<entry xmlns='http://www.w3.org/2005/Atom'
xmlns:gd='http://schemas.google.com/g/2005'
xmlns:gCal='http://schemas.google.com/gCal/2005'>
<id>removed</id>
<published>2006-09-04T18:45:41.000Z</published>
<updated>2007-03-15T23:50:18.000Z</updated>
<category scheme='http://schemas.google.com/g/2005#kind'
term='http://schemas.google.com/g/2005#event'></category>
<title type='text'>Cleaners</title>
<summary type='html'>Recurring Event<br>
First start: 2006-09-01 11:30:00 BST <br>
Duration: 3600   <br>
Who: Shared Calendar  <br>
Event Status:     tentative</summary>
<content type='text'>Recurring Event<br>
First start: 2006-09-01 11:30:00 BST <br>
Duration: 3600   <br>
Who: Shared Calendar  <br>
Event Status:     tentative</content>
<link rel='alternate' type='text/html' href='removed' title='alternate'>
</link>
<link rel='self' type='application/atom+xml' href='removed'>
</link>
<author>
<name>R.I. Pienaar</name>
<email>rip@removed</email>
</author>
<gCal:sendEventNotifications value='true'></gCal:sendEventNotifications>
</entry>

Have a good look at that, this is the default feed output, it shows you details for a recurring event – when it was first created and how long it runs for. There is not enough information here for a human to actually figure out when this event is for, they do not give you any information about the schedule, this event may recur once a year, once a day, once a month you just don’t have any idea. You’ll see later that they have all the ability already to make this useful and user friendly, they just didn’t bother.

The devs in the Google Support Groups say the default feed is for ease of reading by a human and not for machines, for machines you need to use the full feed query format . This lets you do all sorts of good queries for feeds, date ranges and all sorts of things.

Not only is the default feed not machine friendly, no human can figure out what a recurring event is, when it runs etc, as a data feed the default GCal feed is completely useless to everyone.

Using the full feed I now have a daily mail that I have to schedule on my own server to produce a daily agenda, something like this:

Things to do TODAY - Monday, 02 April 2007
No events.
Things to do TOMORROW - Tuesday, 03 April 2007
Test Event
Things to do next 7 days:
Friday Cleaners
Friday Good Friday
Saturday Rebecca Birthday Party
Monday Easter Monday

I had to write this myself using perl and had to go and do this using the full feeds, interestingly enough the full feed has options to expand the recurring events into a single entry, so for instance my Cleaners entry in the agenda above is the same recurring event from the XML earlier. So clearly their backend has the ability to turn the nasty useless recurring event format into something totally human readable and useful, they just chose not to make the default feeds a useful feature.

The calendar has a lot of other questionable features and limitations, it really is a load of compromises, bugs and half baked features cobbled together in something that does not really work as a proper shared calendar environment like a paying customer would expect. And these limitations is not limited to the free versions, they are exactly the same in the paying version. I’d say anyone serious about shared calendar apps or about 3rd party integration should really give GCal and Apps for Domains a miss.

Google Apps for Domains part I

A few of my clients have indicated an interest in moving their mail to Google Apps for Domains, for those living under rocks it’s basically GMail hosted on your own domain. It combines Google Docs (Spreadsheet, Word processor), Calendar, Google Talk and a few other services into one.
So I setup a a test domain a while ago to do a proper evaluation but not against their pay version using the free service. The idea is if the free version is passable but not fatally flawed then the paid version should be fine.
It’s been painful to say the least. First I mistakenly set up a domain that I shouldn’t have since it was already in use and I couldn’t migrate it, my mistake. I figured I’ll be nice and let them know of this and get them to shut it down – a useful exercise since if you do decide to move to them and later want to move away, what you don’t want is your domain in limbo on the Google servers with possible mail routing from inside Google to it and not your real domain.
So first I try and change the primary domain name, can’t, you can add other domains, but it’s impossible to change the primary domain. So I contact them and ask them to shut it down. They contacted me saying no problem, all I need is to send them the PIN code I can get inside the control panel. The thing with these PINs are they keep changing often, so an ex staff member for example can’t claim to be working for you since the PIN won’t be valid anymore, nice feature.
Problem is it takes them longer to respond to support requests than PINs are valid for. So the process goes as follows:

  1. Ask to close domain
  2. Get asked for the PIN, supply the PIN
  3. Support people checks your PIN, it’s not valid anymore because they took too long to check it.
  4. Get asked for PIN, supply the PIN
  5. Go back to 3

That’s just pathetic. Combine this with shoddy level of service, there has been several well published extended downtimes in the first month alone and other weird outages. Yesterday for example I couldn’t access any attachments since some automated bot decided I did something wrong and returned a “Lockdown in sector 6!” error, amazingly descriptive that is, needless to say I never heard from support on this issue.
There are other deep rooted issues as well like with the feeds out of calendars etc, but more on that later. I’ll make a few posts here over the next while with my experiences.