Cayenne Plugin
I have been playing with Shindig to create a datamodel to go behind the OpenSocial API. At first I thought it would be simple to use JDBC direct, but it turns out that the model has a reasonable number of dependencies and the entities are quite big. So, I decided to use Cayenne. The model is created and has some simple test cases, and it all appears to work. I did write a maven plugin to generate SQL scripts from the Cayenne model, so its supports about most major databases (about 8). Creating the model from some Java interfaces was quite easy, infact I did most of it by converting the java class into cayenne XML.
The SQL that this generates at runtime, when querying etc appears much cleaner than Hibernate…. from a DBA point of view.
- https://source.sakaiproject.org/contrib//tfd/trunk/social-db/ The OpenSocial API Data layer
- https://source.sakaiproject.org/contrib//tfd/trunk/social-db/src/main/java/org/apache/shindig/social/opensocial/db/SocialMap.map.xml The Cayenne model
- https://source.sakaiproject.org/contrib//tfd/trunk/maven-cayenne-plugin/ A maven 2 Cayenne plugin that generates the SQL scripts for creating the DB
plexus-util build problems
Looks like there are some dependency issues with building maven plugins. Plexus Utils 1.1 appears to be at fault, with messages such as NoSuchRealmException. If you get this, try upgrading the maven api to 2.0.6 or later in your plugin build. Worked for me.
Widget accessibility and Shindig
I have spent most of the Day playing with Shindig, integrating its data-model into Sakai and learning Guice, which is very nice and simple, especially its error feedback, which just seams to get straight to the problem.
While doing this, on and off I have written a page on Widget Accessibility more because I have been struggling to understand what we really need to do, and want someone to tell me what’s wrong. If you are reading this, and you know all about ARIA and how to write accessible web applications, please tell me what not to do…. but please keep it simple so I can understand what you are talking about.
Offline Maven 2
Ever tried to run maven 2 with no network and Snapshots, after midnight…. its hard because it likes to check for updates of snapshots. The following in ~/.m2/settings.xml might help.
<?xml version="1.0"?>
<settings>
<profiles>
<profile>
<repositories>
<repository>
<id>local-repository</id>
<url>file:////Users/ieb/.m2/repository</url>
</repository>
</repositories>
<id>local-offline</id>
</profile>
</profiles>
<activeProfiles>
<activeProfile>local-offline</activeProfile>
</activeProfiles>
</settings>
Embedding Nasty Templates, Trick
If you have a template that, becuase of its nature/strucutre/purpose is not valid XML… yes it does happen… (sherlog search snippets), you can insert them into a comment and then a CDATA and retrieve them from the DOM. Yes, HTML comments are in the DOM.
render : function(templateName,contextObject) {
if ( ! Template.templateCache[templateName] ) {
var templateNode = document.getElementById(templateName);
var firstNode = templateNode.firstChild;
if ( firstNode && ( firstNode.nodeType == 8 || firstNode.nodeType == 4)) {
template = templateNode.firstChild.data.toString();
} else {
template = templateNode.innerHTML.toString();
}
Template.templateCache[templateName] = TrimPath.parseTemplate(template,templateName);
}
return Template.templateCache[templateName].process(contextObject);
},
nodeType 4 is CDATA, and nodeType 8 is a comment. However, IE5 doesn’t support the node.data property… so I could just do some regex to strip the CDATA or comment tags from converting the element to text.
Using the approach makes it possible to have valid XHTML even with templates for other formats embedded inside the document
Javascript Templates
There are 2 main ways of generating markup from within javascript. Either you perform DOM manipulation, as string injection or direct DOM manipulation. Or you can use templates.
For the java developer, DOM manipulation and string injection is rather like hard coding strings into you java code. templates are more like using jsp, velocity or freemarker.
So I started to look at writing a multi file upload that gave progress feedback, and rapidly found that many of the off the shelf components that did this, didn’t expose the markup, so I had to accept the results as they were, or as I could manipulate them with css. If the markup was wrong, I had to dig deep, sometimes very, into the javascript code.
Then I found TrimPath http://code.google.com/p/trimpath/wiki/JavaScriptTemplates , on Google Code. A templating language for Javascript. It looks a bit like Smarty templating from PHP and has some similarities to Velocity. It works by processing a template, either as a string or as a element from the HTML DOM and merging it with a Tree of javascript objects, just like Smarty and Velocity do.
The result is all the markup is now in the html page, and so can be designed and edited. You can see the results by looking at the code at https://saffron.caret.cam.ac.uk/svn/projects/MyCamToolsAlpha/trunk/files/widgets/DropBox/index2.html
This actually runs from SVN, although some versions of FireFox don’t recognize the CSS files as apache isn’t configured quite right…. Oh… and its trunk, so its going to change as I work on it. (no need to deploy any more, just run Sakai direct from SVN 
)
Lucene Index Merge and Optimisation
Lucene index merge has some parameters that effect how the index is built. This has an impact on the index operations other than search. The MergeFactor controls how many documents are stored within each segment before a new one is started and how many are started before they are collected into a larger one. So a Factor of 10 means, 10 documents before aggregating and 10 aggregated indexes of a certain size before aggregating again. Consequently MergeFactor controls the number of open files.
The higher the merge factor the faster the index build as merging of segments is less frequent. However this causes a significant slow down in the speed which an index can be added to an existing one as this appears to depend on the number of files lucene has to open.
The next one is the MaxBufferedDocs parameter which controls the number of documents to buffer in memory before flushing to disk. For a batch index operation the higher this is the higher the index performance but the more memory will be consumed.
And then there is a MaxMergeDocs which limits the maximum number of documents within a segment above which merging does not happen. This is used to limit the files size, so that no file is over 2G on a 32bit system.
In running the Sakai search indexer operations I have noticed some things in this area
- Once there are about 50 index directories in a merged index, merging takes 2s per merge. Performing an optimize on the index restores the addDirectory operation to 20ms or less. It makes sense to optimize and index when there are more than 50 directories in the index.
- When performing an merge and optimize of a set of indexes, the optimize step can take a lot of time. (minutes). However I have observed that if the index directories are added to an empty index, in the sequence that they were created, the optimize operation is much faster. This may be because the aggregation steps are simpler. This is only an observation.
Installing Sources.
To be able to install jar sources you can run the mvn source:jar maven command and that will put jar sources into your local repo, so you can use them in eclipse.
Reducing Working Code Size
How many of us load the whole of the Sakai Code base into eclipse, and wonder why it consumes so much memory? Most I guess. Alternatively you can just load the code you are working on and just use the local maven repo for the Sakai jars, that way eclipse will run in considerably less memory. When you need to access the source code, if the repo has the source jars, then they can be used instead of the live code base. Obviously this doesn’t allow you to edit all any code anywhere…. but then should we all be doing that anyway… except for those rare debugging exercises.
I did the above for search, editing the .classpath file for eclipse and now I can just work on search with all the other projects close. Eclipse memory usage has dropped from 1G to closer to 128M. Once we package the core (bin and src) into a maven repo, its going to make sense to use this approach. Fortunately maven has support to help us.
Why Spring and Hibernate cant be seperated
After extracting the spring-core, spring-hibernate3 and all the various parts of Sakai, fixing the classloader issues surrounding IdGenerators etc, I find both Spring and Hibernate use CG Lib for proxies, and if you separate Spring from Hibernate, they fight over CGLib. Either Hibernate cant create proxies because it cant see the hibernate classes from CGLib or Spring cant get to CGLib because its not in shared. Looks like its not going to be possible to separate Spring and Hibernate into separate classloaders without providing some extra level of visibility between the classloaders.