Java Applet logging

The last couple of days I have been working on logging from our applet. Frustrating indeed, but finally it works, almost as planned.

Our first approach was to do the configuration programatically, something like:

//get the root logger
Logger logger = Logger.getLogger("");
//create a ConsoleHandler
Handler handler = new ConsoleHandler();
handler.setLevel(Level.FINE);
logger.addHandler(handler);
logger.setLevel(Level.FINE);

Now, this works very well in the Applet Viewer, but when you run your applet in a browser you'll get an exception, since the applet can not change the Handler configuration.

Well, there is the getAnonymusLogger() function, which is designed for use in applets. An anonymous logger would mean that we miss much of the benefits of the flexibel configuration, which is based on named loggers, but I guess it is better than nothing. But while using this method makes it possible to the the level on the Logger, you still can not configure the handler, so you will not get any logging output still.

So, we had to go for the configuration file. This is actually not so bad. You find the logging.properties file (it is under jre.lib) and configure your logging there. For console logging, you have to change the level of the ConsoleHandler (configured by default, but set to INFO level):

java.util.logging.ConsoleHandler.level = FINE


If you want logging to file, you can add the FileHandler instead:

handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

And of course, you turn your logging on:

com.xx.package.level = FINE

Now on my Windows PC this works both in applet viewer and in browser. But when I try the ConsoleHandler on my Mac, there is another gotcha, Bug ID 6585429 LoggerTraceListener causes infinite loop under some circumstances. On my mac the console simply loops and writes the same message over and over again. Seems like this is a known bug, and there should be a fix available, but for now we are stuck with logging to file on the Mac.

Shopping List gadget updated

My shopping list gadget was one of my first, and is still my most popular. It was originally written as an inline gadget, and has been inline all the time, up till a few days ago. But lately I had complaints about problems with it. So this weekend a took a look. It seems like inline gadgets do not work as before anymore. Inline is actually not supported for new gadgets, so this is not a surprise. So I had to rewrite it and change the type. I also removed all the __MODULE_ID__, which are no longer needed, since it is rendered in an IFRAME.

Web site revised

I have spent some hours on revising my private web site jubic.se. The new site is based on css positioning instead of the old one, based on html tables. Much esiser to maintain and hopefully a bit more modern look. Some changes to the content too, mainly on the main page, which is re-written. The site contains motsly material on my Google gadgets. I hope to have the time to add some more the following weeks, and possibly also some MySql material later.

Blogger problem solved!!

At last I manged to get my javascript code into blogger. The problem was with < and > which Blogger don't like. So i had to create small strings and use instead:

var lt = String.fromCharCode(60);
var gt = String.fromCharCode(62);

And now everything works as it should! Check it out at http://rockpopfavorites.blogspot.com/

It is behind the YouTube search at the right, which is actually a Google gadget (not yet published) where parameters are set dynamically.

Blogger hates me.....

I have been trying most of the week to add some HTML and javascript code to a blog. It looks OK in the edit window but every time I save it, Blogger destroys my beautiful javascript code... It tries to format it, moves fragments of code around and destroys it completely. Escpecially special chars (including the not so special " and ' chars and whitespace!!!) seems almost impossible. I have spent several hours of my precious spare time on this.

Applet lifecycle

Lately I have been working with applets. This is new to me, since I have been mostly a server and web guy, with J2EE and servlets as my specialities, and some J2ME. And now I have run into a strange problem....

An applet's life cycle has four methods:
- init() when it is created
- start() when it is shown
- stop() when it is no longer shown
- destroy() for releasing resources

There is an old exception to this, that som browsers (Internet Explorer, possibly some other too) sometimes i9nvokes init() more than once. We had some problems with our applet which seemed to be caused by this. But when it made a fix for this, it did not help...

Some more investigation showed that actually Internet Explorer did not only invoke init() twice but also created two instances and invoked both init() and start() on both. This seems to happen the first time an applet is run, but possibly not the second. Very strange indeed....

Create thumbnail in J2ME

The latest week I have been working on a J2ME project where we need to display thumbnails of piuctures from the mobile camera. I suppose this is a common scenario, but we had some unexpected problems..

The main reason for the problem is that mobile cameras today are very good. This means that images often will be very big, too big to display, but also to big to handle as an image in the J2ME environment. If you try Image.createImage() you will get an out-of-memory exception. This means that you can not create an Image even to create a thumbnail. In stead you have to create the thumbnail image straight from the stream, without creating a full-size image first.

There are different ways to do this, but since we work with a target platform that is quite new, and supports JSR-234 we chose to use this feature. A small utility function to do the resize could look something like this:

public static Image resizeImage(InputStream src, int width, int height)
{
 MediaProcessor mp;
 try {
    mp = GlobalManager.createMediaProcessor("image/jpeg");
    mp.setInput(src, MediaProcessor.UNKNOWN);
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    mp.setOutput(bos);
    ImageTransformControl fc = (ImageTransformControl) mp
        .getControl("javax.microedition.amms.control.imageeffect.ImageTransformControl");
    fc.setTargetSize(width, height, 0);
    fc.setEnforced(true);
    fc.setEnabled(true);

    mp.complete();
    src.close();
    return Image.createImage(bos.toByteArray(), 0, bos.size());
  } catch (MediaException e) {
      System.out.println("MediaException in Utils.resizeImage():" + e.getMessage());
  } catch (IOException e) {
      System.out.println("IOException in Utils.resizeImage():" + e.getMessage());
  }
  return null;
}

RSS Reader Gadget updated

I have made a minor update to the RSS reader gadget. You can now set how old items that should be displayed. Increases usability quite a lot.
Check it out here.

Web enabling MySQL

Mark Atwood has published an interesting post with his wishlist for MySQL. One of his points is already (partly) covered with the LOAD XML command I made for MySQL 6.0 (or is it 5.2).

The main problem with it is perhaps that it works (like the LOAD DATA command) only with files, but in a web world you want it to be able to take input from other sources. You also need to be able to send an XML to several tables, with foreign keys coordinated etc.

Mark also has a wishlist for the opposite, the SELECT OUTFILE command. I have made a couple of stored procedures to help with this, but an extended SELECT OUTFILE would be mucke aesier to use and give much more possibilities. It would be fun to work on this, we'll see if I have the time, now that I am almost ready with my relocation..

RSS reader gadget

For a while I have been using a small PC program for the feeds I keep a look on. It has lot's of functionality, and with some work I can get it to do just what I want to, which is very simple: just display a list of my feeds, where I can expand a feed and look at the posts.

But this gave me some problems... Sometimes it crashed, and I loose all my settings. I then have to set up everything again, which is a lot of work. Also I only had it on one of the computers I use (three in all), which meant I could only follow the feeds on that computer.

So the last time it crashed for me, i deinstalled it and made a Google gadget for feeds instead. So far it is very basic, the main advantage over the ones I have seen is that it handles multiple feeds ( there are excellent gadgets for handling only one feed). Probably I will add more options in the future.

Back to the J2ME world

I have got a new assignment, for a J2ME midlet this time. Primary target is not a small cellphone, but the new BlackBerry Curve. This means we have to think differently, the BlackBerry has a muck better interface, larger screen and a qwerty keyboard. We have to make sure they are used the best way possible. My previous J2ME experience has been with cellphones from Nokia, Sony Ericsson and Motorola, this is something different.

The BlackBerry environment means that additional preprocessing has to be done. Not a big thing, but yet another point that differs from PC/Windows client, where you just develop form Windows (possible verify on both XP and Vista) and don't care about the hardware. The J2ME platform is still not that mature (will it ever be?), you have to verify on the major phone brands, both that it works and looks OK.

Import and Export

Recently we have been doing some work with export and import of data from and to MySql databases. Our customer wants to be able to export configuration data from one MySql instance and import it into another.

The obvious tool to use is mysqldump. But since we need to edit the exported data (just replace some codes) before importing it into the other instance we use the --tab option. This means that instead of one big file (or actually redirected standard output) we get two files per table, one with the CREATE TABLE command and one with the data.

In the next step we run some scripts to modify the data. This is quite easy, since data is in text files with one record per row.

Finally we import the data into the new instance. While the default mode of mysqldump creates a file you can run in the command line client, mysqldump --tab does not. You have to create that yourself, but it is not very complicated. You need the following lines for every table:

source [tablename].sql
LOCK TABLES [tablename] WRITE;
ALTER TABLE [tablename] DISABLE KEYS ;
LOAD DATA INFILE '[path][tablename].txt' INTO TABLE [tablename] ;
ALTER TABLE [tablename] ENABLE KEYS ;
UNLOCK TABLES;

The ALTER TABLE and LOCK/UNLOCK commands are not absolutely necessary, but they speed things up. We have not measured the speed of this approach, but the general impression is that it is fast.

Probably this method is faster than the default mode of mysqldump, because data is not sent through the parser, where there has been some performance problems. LOAD DATA INFILE is in most cases very fast.

Out of memory......

Last week or so has been busy. We are working on our next release, which seems to work well on a functional level. But we get problems with Out of memory errors...

There are different causes for this, but the main cause is with Hibernate. The part that causes the most problems is where we read quite a lot of data and generate data for the online part of the system. The data is stored as BLOBs in the database, and Hibernate is used througout. We want to be sure that everything or nothing is processed, so it is all done in one tranmsaction.

Now it seems that even though we do not read from the table with the BLOBs, only write to it, Hibernate keeps all records in the session cache, filling the memory. To me this seems very strange. We dont need this cache, once the records are written to the database, they should be removed from the cache. In my experience this is a very common scenario, that you do not read the records created again in the same process. In fact it is good practice to get all data before inserting, so you do not need to read the same record again.

It is also difficult to understand that Hibernate does not take care of Out of memory conditions. A good cache should not grow without limit, it should purge data when memory usage is too high. This is not something you should have to do in your application.

The problem seems to be solved for now, but since data volumes in the system will grow, it will probably reappear. We will have to do something more permanent for the future. Probably this will be to bypass Hibernate for the BLOB insert, and do this with JDBC calls.

Moving !!

I haven't had much time for this blogging for a while, and probably won't have until perhaps in december. Most of my spare time is spend on moving.

After 23 years I am leaving Lund. Not going too far though, just some 60 kilometers, to Helsingborg. From the University town of Lund, where Sony Ericsson is the largest employer, to the harbour town of Helsingborg, with IKEA:s IT department.

Hibernate proxy classes and reflection

I have been struggling the last few days with a small tool to export and import data from a system using Hibernate Object Relational mapping. To make it fairly general, and avoid having to change it when attributes are added to the data model we based it on Java reflection. Fairly straightforward, create an XML file with classnames as tagnames and fields as attributes.

The first problem was object relations. My initial plan was to simply follow them, but sinces they are all bi-directional this would mean looping forever. So we had to add a setup where you tell the tool which object relations should be included.

The next problem was worse. It seems that Hibernate somtimes does not create the right objects when loading objects from the database. Instead so called 'proxy classes' are created. This means that reflection does not work as expected... You cannot use Field to acces the fields, instead you have to use the getter methods. OK, not to complicated, once you know. Unfortunately we had one attribute which did not follow naming standards, so we needed a small hack to make this work. What I would have liked is a way to get the real class and skip the proxy. But i guess you can't get everything.

SQL mode quiz answers

My quiz about SQL mode was a bit more difficult than I expected. The question was which ones of the following statements are affected by a change from the default SQL mode to ANSI mode.

I used the INFORMATION_SCHEMA tables for examples throughout. This is to make it easy to try for yourself, without needing to create tables etc. If you use the Query Browser to try this out you should beware that setting the sql_mode might not work, since the Query Browser by default creates a new database connection for every statement. The easiest way to check this si to use the 'set global sql_mode' command, but you should not do this on a production database, since strange things might happen....

If you want to read the documentation, you find it here.

Now for the solution...

a) SELECT concat(table_name,' ',column_name) FROM information_schema.COLUMNS

This was quite an easy one. CONCAT is a MySQL extension to ANSI SQL, but it is not disabled by the ANSI SQL mode. The command will work just as well whatever the SQL mode. But CONCAT is not portable to other databases. Oracle has a CONCAT function, but it takes only two arguments. SQL Server has no CONCAT, and I don't know any other databases that has one.

b) SELECT TABLE_NAME||' '||COLUMN_NAME FROM information_schema.COLUMNS

That was also an easy one. Double pipes is the ANSI standard way of string concatenation. In MySQL it means OR.

So, in ANSI mode you will get:

+-------------------------------------+

| TABLE_NAME||' '||COLUMN_NAME |

+-------------------------------------+

| CHARACTER_SETS CHARACTER_SET_NAME

| CHARACTER_SETS DEFAULT_COLLATE_NAME

| CHARACTER_SETS DESCRIPTION

etc

In default mode:

+------------------------------+
| TABLE_NAME||' '||COLUMN_NAME |
+------------------------------+
| 0 |
| 0 |
| 0 |
| 0 |
| 0 |

c) SELECT count (*) FROM information_schema.COLUMNS
This was a tricky one. I guess it was not easy to see, but there is actually a space between count and the left parenthesis. This makes it invalid in MySQL standard mode:

mysql> SELECT count (*) FROM information_schema.COLUMNS;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use near '*) FR
OM information_schema.COLUMNS' at line 1

Change to ANSI mode, and it works:

mysql> set sql_mode = 'ANSI';
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT count (*) FROM information_schema.COLUMNS;
+-----------+
| count (*) |
+-----------+
| 503 |
+-----------+
1 row in set (0.11 sec)

So, this is a statement that works differently....

d) SELECT "TABLE_NAME" FROM information_schema.COLUMNS
This was also quite easy. Double quote is according to ANSI standard used for identifiers like table and column names. You only need them if you have characters that would otherwise be illegal or if you want to have mixed case in your identifiers.

By default MySQL uses double quotes for strings. This means that the select will give the following result:

+------------+
| TABLE_NAME |
+------------+
| TABLE_NAME |
| TABLE_NAME |
| TABLE_NAME |
| TABLE_NAME |

If you change to ANSI, you will get the following result:

+----------------+
| TABLE_NAME |
+----------------+
| CHARACTER_SETS |
| CHARACTER_SETS |
| CHARACTER_SETS |
| CHARACTER_SETS |
| COLLATIONS |

e) SELECT `TABLE_NAME` FROM information_schema.COLUMNS
This is the MySQL alternative to ANSI double quote. It does not follow ANSI standard, but the MySQL ANSI mode will not turn it off. I do not know another database that allows this syntax, so if you try to convert your system you will have to remove them.

Conclusion
Statements b, c and d are the ones that will behave differently if you change to ANSI mode. Statements a and e will work the same.

SQL mode quiz

MySQL follows ANSI SQL standard quite close. But there are a few points where it differs. To make MySQL follow the ANSI standard more closely you can use the SQL_MODE system system variable.

If you set the SQL mode to ANSI, MySQL will interpret you SQL commands differently. Some of your SELECT statements might work differently or not at all. But which ones? Let's see if you know....

Suppose you change your sql mode setting from none to ANSI. Which of the following statements will be affected:

a) SELECT concat(table_name,' ',column_name) FROM information_schema.COLUMNS
b) SELECT TABLE_NAME||' '||COLUMN_NAME FROM information_schema.COLUMNS
c) SELECT count (*) FROM information_schema.COLUMNS
d) SELECT "TABLE_NAME" FROM information_schema.COLUMNS
e) SELECT `TABLE_NAME` FROM information_schema.COLUMNS

Follow up on LOAD XML

A few weeks ago (time flies..) i posted a short note about my LOAD XML contribution. It has actually been available for some time, veven if it is not so easy to find. It is included in MySql 5.2.5:
download
documentation - quite good I think!

It is a bit sad that it is included in 5.2 (still in alfa) and not in 5.1. Since it is a separate command it will not affect existing installations and would not be risky. That faster it gets included the faster we will get feedback on it, which is what I believe is needed right now. Is this something that MySql users need and will use? Or will they use other tools to read XML data into their MySql databases?

LOAD XML reads data into a MySql table in much the same way as LOAD DATA does and in fact the implementation is based on LOAD DATA. This means that it does not parse the entire XML document ( I belive MySql xml function ExtractValue() does that) but instead reads one row at a time from the xml file and inserts it into the database. This makes reading of large xml files possible at quite a decent speed, althoug not as fast as LOAD DATA.

The mapping between XML and the table is based on two assumptions:
- there should be an XML tag that corresponds to a record in the table. You name this tag in the 'ROWS IDENTIFIED BY' clause
- both tags and attributes are matched automatically with column names and values
inserted into the tables. Tags and attributes that do not match columns are simply ignored

When I implemented this the scenario it was designed for is an integration scenario, where you receive data in an XML file periodically and dump it into your database.

There are of course limitations you should be aware of:
- CDATA is not supported and binary fields can not be handled
- it can only load data from a file
- the XML is not verfied, an illegal XML could very well be processed, no validation with xml schema or DTD is made
- there is no namespace support

All in all I think this could be quite a useful function. Arnold Daniels, who worked with me on this contribution initially, is planning some improvements. Among other things to make this work with XML data that is not read from a file.

Personally I will perhaps make some minor fixes, but probably start on something entierly different. Don't know what yet, your suggestions are welcome....

LOAD XML contribution added to MySQL

I got news from the MySQL people that my LOAD XML contribution is added to release 5.2. Check the manual page.

The aim of the contribution is to simplify reading of XML data into a MySQL database. The LOAD XML command supports three different xml formats:
- field values as attributes
- field values as tags
- the format produced by the mysql -x command, with the field name as an attribute and field value as a tag

If the LOAD XML command finds a matching field in the target table, the value is inserted, otherwise it is ignored. If you have used the LOAD DATA command (most of us have) you should recognize much of the functionality, LOAD XML works much the same way.

More on XML output from MySql

In a previous post I described how you could create XML output from SQL using stored functions. But that was a flat structure, with only one table and simple XML structure with a tag for every row in the result set, and column values as attributes or sub tags. Part of the real power of the XML format is that it supports more complicated structures, with hierarchical data. If you need to produce XML for integration with another application you will probably have to produce a XML with data from several tables, and a hierarchical structure. To accomplish this in MySQL we can use the GROUP_CONCAT function, combined with our XML stored functions.

An example:
We take our select statement from the previous post, that gives us a list of tables in xml format:

select xml_tag('table',null, concat(xml_attr('name',t.table_name),
xml_attr('engine',t.engine)),null)
from information_schema.tables t
where t.table_schema = 'INFORMATION_SCHEMA'

We then add a join with the columns table to get column data:

inner join information_schema.columns c on c.table_name = t.table_name and c.table_schema = t.table_schema

and replace the last parameter of the xml_tag function call, the one for the subtags with a new xml_tag function call to produce a column tag:

xml_tag('column',c.column_name,null,null)

This gives us:

select xml_tag('table',null, concat(xml_attr('name',t.table_name),
xml_attr('engine',t.engine)),xml_tag('column',c.column_name,null,null))
from information_schema.tables t
inner join information_schema.columns c on c.table_name = t.table_name and c.table_schema = t.table_schema
where t.table_schema = 'INFORMATION_SCHEMA'

Running this gives us:

<table name="CHARACTER_SETS" engine="MEMORY" >
<column>CHARACTER_SET_NAME</column></table>
<table name="CHARACTER_SETS" engine="MEMORY" >
<column>DEFAULT_COLLATE_NAME</column></table>
<table name="CHARACTER_SETS" engine="MEMORY" >
<column>DESCRIPTION</column></table>
<table name="CHARACTER_SETS" engine="MEMORY" >
<column>MAXLEN</column></table>
<table name="COLLATIONS" engine="MEMORY" >
<column>COLLATION_NAME</column></table>
etc...

OK, this is some kind of structure, column tags inside the table tag. But still we have one table tag for every column, but we want one for every table, with all the column tags inside. To accomplish this we add a group by clause, and use the GROUP_CONCAT function for the column tags. This gives us the following SELECT:

select xml_tag('table',null, concat(xml_attr('name',t.table_name),
xml_attr('engine',t.engine)),GROUP_CONCAT(xml_tag('column',c.column_name,null,null) separator ''))
from information_schema.tables t
inner join information_schema.columns c on c.table_name = t.table_name and c.table_schema = t.table_schema
where t.table_schema = 'INFORMATION_SCHEMA'
group by t.table_name, t.engine

And the following result:

<table name="CHARACTER_SETS" engine="MEMORY" >
<column>CHARACTER_SET_NAME</column>
<column>DEFAULT_COLLATE_NAME</column>
<column>DESCRIPTION</column>
<column>MAXLEN</column>
</table>
<table name="COLLATIONS" engine="MEMORY" >
<column>COLLATION_NAME</column>
<column>CHARACTER_SET_NAME</column>
<column>ID</column>
<column>IS_DEFAULT</column>
<column>IS_COMPILED</column>
<column>SORTLEN</column>
</table>
etc

OK, this is what we wanted! Just one small problem left... When I run this on my laptop I get the error 1260 with the following message:
%d line(s) were cut by GROUP_CONCAT()
And the result is only 7 rows. This is because the group_concat_max_len has a value that is too small. You can reset it with the command:

SET [SESSION | GLOBAL] group_concat_max_len = val

I set it with the command:

SET GLOBAL group_concat_max_len = 2048

I then rerun the command, and there is no error, and 23 records.

Code for the stored functions is available at MySql forge.