Saturday, June 30, 2012

GORM - False Sense of Simplicity

While maintaining some code recently, I came across what seemed to be some particularly evil code.  It was a service method that used a Grails Criteria Query to fish out some objects from the DB.  These were then converted to Data Transfer Objects (DTO) to be serialized for use in a GWT client.  It went something like :


def myObjects = Domain.findAllBySomething(thing)
....
....
....
myObjects.each{obj ->
  def dto = new Dto()
  dto.property = obj.innerDomain.property
  dto.propertyTwo = obj.innerDomain.anotherDomain.property
  dtos << dto
}

It seems fairly innocent but try telling that the database.  Each call on any of the properties within obj that are themselves domains resulted in another database call.  Creating the DTO above resulted in about 3 queries just to set two properties.  This might not sound too bad, but imagine if myObjects contains 1,000 or so objects.... not the best strategy.  The actual code I was maintaining was typically returning about 1,000 results and this resulted in about 13,000 queries running on the database.  This was a user driven event so it could be happening quite a lot.  This was obviously terrible for performance.  Of course there is an argument to use paging and caching of results etc. but that's a different blog post.  Long story short, a colleague (who I won't name for social reasons) and I were able to get the 13,000 queries down to 25 for the entire service method by just configuring the query correctly.

The main point here is the difference between eager and lazy fetching.  A long time ago, Grails' default fetching strategy was eager.  This basically meant that when you queried for domains, using dynamic finders or criteria, your object model would return fully populated.  Unfortunately this could also result in you returning the entire database in one query.  Not good.

Around about Grails 1.0.4,  the default fetching strategy was changed to lazy.  This meant that your object model returned partially populated, that is, proxy objects with only their id's, were returned for properties that were themselves domains.  When you called for a property on one of these proxies, Grails would request it from the database.  All this happened pretty transparently to the user.

Grails provides many ways of forcing earger fetching.  It can be done in the domain mappings and in the domain itself by setting the fetch mode.  It can also be done at a query level which I am more in favour of.  Setting on the domain itself is too much of a commitment in my opinion as you can never imagine all use cases at time of coding.  I will illustrate setting it in dynamic finders with the following domains :


class Customer {

  String name
  Account account

  static constraints = {
    account unique: true, nullable: false, blank: false
    name nullable: false, blank: false
  }
}

class Account {
  static belongsTo = [owner: Customer]

  Long accountNumber

  static constraints = {
    accountNumber min: 0l, nullable: false, unique: true
  }
}

class BankingTransaction {

  Long transactionNumber
  BigDecimal amount
  Account fromAccount
  Account toAccount

  static constraints = {
    transactionNumber min: 0l, unique: true, nullable: false
    fromAccount nullable: false, validator: {Account thisFromAccount, BankingTransaction thisTransaction ->
      if (thisFromAccount?.accountNumber == thisTransaction?.toAccount?.accountNumber) {
        return false//cannot have a transaction to the same account... probably
      }
    }
    toAccount nullable: false
    amount nullable: false
  }

  BankingTransactionDTO toBankingTransactionDTO() {
    def dto = new BankingTransactionDTO()
    dto.with {//probably better ways of doing this but this illustrated the example
      transactionNumber = this.transactionNumber
      amount = this.amount
      fromCustomersName = fromAccount.owner.name
      toCustomersName = toAccount.owner.name
    }
    return dto
  }

}

So to illustrate the lazy and eager fetching I wrote some tests that were bootstrapped on a MySQL database.  First we need to set up some test data and an additional Grooy Sql connection to run some status queries on our DB.


class BankingTransactionTests {

  Sql sql
  SessionFactory sessionFactory
  BankingTransactionService bankingTransactionService

  @Before
  void setUp() {
    BankingTransaction.withNewSession {session ->
      100.times {num ->
        new Customer(name: "Customer ${num}", account: new Account(accountNumber: num)).save(failOnError: true)
      }
    }

    BankingTransaction.withNewSession {session ->
      final allCustomers = Customer.list()
      def generousCustomer = allCustomers[0]
      allCustomers[1..-1].eachWithIndex {customer, index ->
        new BankingTransaction(fromAccount: generousCustomer.account, toAccount: customer.account, transactionNumber: index, amount: index * 100).save(failOnError: true)
      }
    }
    sql = new Sql(sessionFactory.currentSession.connection())
    sql.metaClass.getNumberOfSelects = {->
      return delegate.rows('SHOW STATUS LIKE "Com_select"')[0].Value as Long
    }
  }

....
}

I won't go into the details, it is just an integration test, the code is on GitHub anyway for the forever curious.  So first off lets do a simple test that queries for all BankingTransactions.


  @Test
  void testSimpleQuery() {
    //get the current number of selects in this session
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
    final transaction = BankingTransaction.list()//1 query

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest

    //is it what we expect??
    assert numberOfSelects == 1
  }
We run the test, and huzzah, test passes. As we would expect BankingTransaction.list() is a simple select query.

OK, now lets access some properties on the first BankingTransaction


  @Test
  void testSimplePropertyAccess() {
    //get the current number of selects in this session
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
    final transaction = BankingTransaction.list()[0]
    def fromAccountNumber = transaction.fromAccount.accountNumber
    def fromAccountCustomerName = transaction.fromAccount.owner.name

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest

    //is it what we expect??
    assert numberOfSelects == 1
  }

Uh oh, test fails with :

Assertion failed: 

assert numberOfSelects == 1
       |               |
       3               false


Looking closely at the test we can see the three queries, the first is the the call the list(), the second is the call to fromAccount and the third, the call to fromAccount.owner.

OK, so, lets fix the query by setting the fetch mode to join in the query.


  @Test
  void testSimplePropertyAccess() {
    //get the current number of selects in this session
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    def queryParams = [fetch: [fromAccount: 'join', 'fromAccount.owner': 'join']]
    //carry out the query
    final transaction = BankingTransaction.list(queryParams)[0]
    def fromAccountNumber = transaction.fromAccount.accountNumber
    def fromAccountCustomerName = transaction.fromAccount.owner.name

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest

    //is it what we expect??
    assert numberOfSelects == 1
  }


Test passes! Sweet! This is because we told GORM to fetch the associated domains using a join. I suppose we should look at a more real work example. Suppose we have a BankingTransactionService with the following method :
class BankingTransactionService {

  Collection retrieveTransactions() {
    def transactions = BankingTransaction.list()
    return transactions.collect {it.toBankingTransactionDTO()}
  }

}


With the following test :

  @Test
  void testBankingService() {
    //get the current number of selects in this session
    def numberOfQueriesBeforeTest = sql.numberOfSelects

    //carry out the query
    def dtos = bankingTransactionService.retrieveTransactions()

    //now calculate the number of queries
    def numberOfQueriesAfterTest = sql.numberOfSelects
    final numberOfSelects = numberOfQueriesAfterTest - numberOfQueriesBeforeTest

    //is it what we expect??
    assert numberOfSelects == 1
  }


We run the test, and eh... not looking good :

Assertion failed: 

assert numberOfSelects == 1
       |               |
       201             false


Hmmm 201 queries, this will not please the performance junkies at all. What is happening in toBankingTransactionDTO() is that as the objects are created a query happens each time a property that is a domain is accessed. These slowly build up in the first level cache reducing the amount of queries, but I think we can do better.

Let's go all out and reduce the number of queries to 1 for this service method.  It's gonna be a long road.... not really :


  Collection retrieveTransactions() {
    final queryParams = [fetch: [
            fromAccount: 'join',
            toAccount: 'join',
            'toAccount.owner': 'join',
            'fromAccount.owner': 'join']
    ]
    def transactions = BankingTransaction.list(queryParams)
    return transactions.collect {it.toBankingTransactionDTO()}
  }


Test passes. There we go, 201 queries to 1. Not really that much work, just don't take the simplicity provided by GORM for granted or you will have some angry DBA's on your tail... you have been warned!

Slán

Code on GitHub.

Monday, January 24, 2011

Quick Groovy Script! Google Geocoding!

Nice little script for parsing the Google Ceocoding Web Service. (Groovy 1.7.5)


/**
Simple Script to parse the google geocode web service
http://code.google.com/apis/maps/documentation/geocoding/
*/
def url = new URL('http://maps.googleapis.com/maps/api/geocode/xml?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&sensor=true')
def geoCodeResult = new XmlParser().parseText(url.getText())
def lat = geoCodeResult.result.geometry.location.lat.text()
def lng = geoCodeResult.result.geometry.location.lng.text()
println lat
println lng


Outputs :
37.4216227
-122.0840263

Monday, November 22, 2010

Grails Quick Tip : External Configs

A pretty decent bloke once said that "Simplicity is the ultimate sophistication".... and that is what I like about this idea; it is so damn simple and the rewards gained from it are vast. Well maybe not vast.... but there's a few. The idea I am harping on about is, external configuration files in Grails.

Ok so, the sales pitch. You have created the worlds greatest application, (not difficult with Grails) and you are on top of the world! Then suddenly, the emails you have configured to send out every minute to remind you how great you are, have stopped! What the???? The mail configuration settings have changed???? Uh oh, we are going to have to redeploy the application?? I am in the airport! Uploading my 30MB war file is going to take hours... weeks.... DAYS! I haven't tagged the code and the war might contain untested code!!!!

Relax... breathe... there is an alternative!

External configuration files in Grails allow you to extract the tidbits from Config.groovy that you feel are important. Things like the mail configuration that can change at any time. Others like your datasource password which you don't really want checked into git, subversion or whatever version control you may be using!

Enabling external configuration files in your Grails application is as easy as uncommenting a few lines of code in the top of your Config.groovy, the one's that look like this :


// grails.config.locations = [ "classpath:${appName}-config.properties",
// "classpath:${appName}-config.groovy",
// "file:${userHome}/.grails/${appName}-config.properties",
// "file:${userHome}/.grails/${appName}-config.groovy"]


For this example, let's work with this simple case :


grails.config.locations = ["file:${userHome}/.grails/${appName}Config.groovy"]

Here we have defined an external config that will sit in a hidden grails folder in the user's home directory. userHome is a variable added to the binding of Config.groovy at runtime and works on windows and unix based boxes alike. Similarly, appName is also a variable that is added to the binding at runtime containing the name of your Grails app.

This is all well and good but how does it work in the real world??? A very real scenario is that you will want to use external Config files in production and use the regular Config.groovy during development. No one wants to navigate outside their Grails application directory during development, we are not savages after all!

So..... how do we configure grails to look at an external configuration file in production and still have the flexibility of using Config.groovy on development?? Well you can use the environments closure in Config.groovy like so :


environments {
production {
//all my production configs are externalized
grails.config.locations = ["file:${userHome}/.grails/${appName}Config.groovy"]
}
development {
//the dev environment is none the wiser!
grails {
mail {
host = "mail.server.com"
'default' {
from = "mail@me.net"
}
port = 25
username = "mail@me.com"
password = "password"
props = ["mail.smtp.auth": "true",
"mail.smtp.socketFactory.port": "25",
//"mail.smtp.socketFactory.class":"java.net.ssl.SSLSocketFactory",
"mail.smtp.socketFactory.fallback": "false"]
}
}
}
test {
grails.gorm.failOnError = true
}
}


You can also declare settings that apply to all environments outside of this closure. This has the added bonus of reducing the 'noise' in your production config which should be kept as clean and as crisp as possible.

Now thinking back to our imaginary scenario at the beginning, our jet set developer can now just log into the server, change the offending setting and bounce the server! (Another step would be to database the settings to prevent bouncing but this is slightly off topic.)

It is also possible to refactor your dataSource config options to your external file... something along these lines will work :



dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
username = "root"
password = "password"
url = "jdbc:mysql://localhost/pretty_db"
}


Huzzah! Now if you need to change db password, you do not need to redeploy a new war!

Honestly, one of the more convenient advantages of doing this is the ability to work with the logging of your application in production. This can be a pain when you first deploy to an app server on a production server away from the cushy dev environment that Grails provides!

It is worth noting too that you should set your external files up with permissions of 600, the owner of the file being the user that you run tomcat or your other container with.

So one simple change has got us :
  • Ability to change Config options without having to build a war
  • Separated our Datasource options from our application... I assume the DBA's will thank you!
  • Tuning and changing logging has now become a lot easier
  • Your war can now be transfered to different servers with different file systems and all you have to do is change a config file...
In these recessionary times, you cannot argue with that value! Thanks to Tim Berglund for showing how easy this was at his talk at SpringOne2Gx this year :)

This is a great post on using environment variables to achieve this :


UPDATE : It looks like someone has created a plugin that allows you to reload the external config without a restart (I have not used this but should be reliable :) ):


Hope this saves someone the hassle of re-warring for silly reason... it is quite the pain!

John :)

Friday, July 30, 2010

Grails Quick Tip : Work Directory

Yesterday I was required to work on a branch of one of our projects and came across a little predicament : The branch had different plugin versions than the trunk I was currently working on.

Grails stores all the plugin classes, files and resources in ~/.grails, the projects are then divided out by their Grails version and project name. In this case the Grails version and of course the project names matched. I was a little peeved that I might need to blow away plugins and re-download if I was switching branches etc. What is one to do at a time like this??? Cry??? No..... just add the following line to BuildConfig.groovy under the grails-app/conf directory :

grails.work.dir="work"

This will simply place all your project meta-data into your project base dir /work... No need to blow away your previously downloaded plugins. Tears averted! I think this was introduced in 1.1.

One cool plus is that Intellij IDEA picks up on this (as with a lot of changes in BuildConfig.groovy these days) and allows you to continue with no pesky configuration changes! Also, this folder doesn't need to be in your grails-app, it can be anywhere, just use absolute paths.

Hope this help avert some crying developers.... they aren't much fun (except when Captain Morgan is the cause)

:) Thanks!

Monday, May 31, 2010

Relationship Advice : Grails One-To-Many

If you are here investigating bigamy.... you have come to the wrong place.... We are actually going to have a quick once over on One-To-Many relationships in Grails and GORM (Let's abbreviate to OTM for the sake of my fingers). Here's what we'll cover:

  • Starting a Relationship
  • Living and Working With Your Relationship
  • Finding out more about your Relationship (or querying :) )
  • What if you don't belongTo someone?

Starting a Relationship :

First of all, what is an OTM? Let's let the code do the talking:

class Parent{
static hasMany = [children:Child]
String name
}

class Child {
static belongsTo = [parent:Parent]
String name
}


Here is a very simple domain model that will serve as a nice concise example. We have a Parent domain that can 'haveMany' children... in turn these children belongTo a specific parent. This is quite a 'tightly coupled' OTM. There can be a few nuances that catch people out here, which of course we will cover... (we will take a brief look at the "loosely coupled" OTM too).

The definition of the hasMany in the Parent class is actually a Map, and it should read like [collectionName:TypeContainedInCollection], so the parent has a collection with the name 'children', that will contain instances of the Child domain class.

Similarly in the Child class, the 'belongsTo' property is also a Map, [instanceOfOwningClass:typeOfOwningClass]. The 'belongsTo' property forms what I referred to earlier as a 'tightly coupled' OTM. This basically entails that the children instances are 'owned' by their 'parent', this in turn has some interesting side effects that we will cover now.

Living and Working With Your Relationship :

OK let's create a parent with some children :

def parent = new Parent(name:'Dad')

//add a new child instance
parent.addToChildren(new Child(name:'son'))

//add another child instance using a map
parent.addToChildren([name:'daughter'])

//save the parent
parent.save(flush:true,failOnError:true)

To add a Child to a Parent we use the 'addTo' method that is provided by Grails, it is basically addTo+Name of Association. When we use these methods, GORM will wire up our domain classes and because children have a 'belongsTo' property set, calling save() on the parent will also validate and save the children... sweetness :)

OK now... what if we need to remove some children from our relationship.... easy....

//retrieve our parent
def parent = Parent.findByName('Dad')

//retrieve the child instance
def child = Child.findByName('son')

//remove the child
parent.removeFromChildren(child)

parent.save()

Oh.... wait.... NOT EASY!!! If we run the above code we get the following Exception:

"not-null property references a null or transient value: Child.parent"

Yeeeeeeeeesh.... not pretty... The knee jerk reaction here might to add a nullable constraint for the parent property in the Child class... but this would be wrong. The relationship we defined has a 'belongsTo' so in this case, if we remove a child.... we simply want it to be deleted automatically at the same time. To achieve this, our Parent domain must look like:


class Parent{
static hasMany = [children:Child]
String name

static mapping = {
children cascade: "all-delete-orphan"
}
}

Now if we re-run the previous code, the child instance will be removed from our relationship and also deleted!


Finding out more about your Relationship :

If we have a Parent instance, it is rather easy to query for our children.... we simply access them as a property:

def parent = Parent.findByName(name:'Dad', [lazy:[children:false]])
parent.children.each{//do something}

However, things are not so clear cut when coming from the other direction. Suppose we have the name of a child and we want to find all the parents that have children with that name, so basically querying the relationship from the many side:

def name = 'son'

//create criteria so we can perform a criteria query
def c = Parent.createCriteria()

//perform a listDistinct as querying relationships can result in duplications
def parents = c.listDistinct{
createAlias("children", "c")
eq("c.name", name)
}

Thanks to Robert Fletcher for blogging on this (link to follow)....

What if you don't belongTo someone? :

In brief, if you define an OTM without declaring a 'belongsTo' property as we have, you're Child instances can exist on their own, without the need of being attached to a parent. One side effect of this is that the cascading save behaviour we saw before does not apply here... you will need to call save() on each domain separately.

So there we have it.... defining, creating and querying a One-To-Many relationship in Grails... I hope this has been helpful to anyone that comes across it! Happy trails!

Resources :
http://adhockery.blogspot.com/2009/06/querying-by-association-redux.html
http://www.grails.org/GORM+-+Defining+relationships

Constraining across relationships:
http://johnrellis.blogspot.com/2009/09/grails-constraints-across-relationships.html

Monday, March 8, 2010

Grails inplace plugin tip - BuildConfig.groovy

Hey... it's late... so lets keep this one short!

(Update: check out this article too! )

I have taken it upon myself to get some re-usable functionality out of our companies grails apps and turn them into plugins.... I thought this was a genius idea but alas I wasn't the person to come up with it... creating grails plugins to modularize your applications is common practice these days.

One headache I came across immediately however was developing an application that depended on a plugin that was also in flux... lets call them MyApp and my-plugin....

I found myself going nuts.... I would come across an error or a shortcoming in my-plugin code while developing MyApp (yes... i am man enough to admit it!!) and had to change the code in my-plugin, do a grails clean.... followed by a grails compile... followed by a grails package-plugin..... ..... FOLLOWED BY ..... a grails install-plugin in MyApp directory... followed by a soul crushing 'y' to confirm installation.... HOLY KEYSTROKES BATMAN!!!!!!!! sorry...

Then I came across a nice solution while pestering Peter Ledbrook on his blog on how to best test plugins....

He pointed me to BuildConfig.groovy in the grails-app/conf folder of MyApp... in there I put the magic words:

grails.plugin.location."my-plugin" = "../my-plugin"


This is basically the path to my-plugin relative to MyApp. I have both in the same workspace directory.

By doing this you only need to package your plugin once to ensure all files are generated.

Now if I find I need to change my-plugin while working in MyApp i simply jump to the source in another IDEA window, make the changes and then restart MyApp.... happy days!!

One word of warning though... I had to remove my-plugin from application.properties as from what I could gather... grails didn't know this reference would then relate back to the one BuildConfig.groovy.... this resulted in my-plugin being deleted.... thank the higher beings for version control!!!

Hmmm now I need to figure out how to do a plugin repo for our internal plugins.... another late night maybe....

Hope this helps.... thanks to Peter for his time!

Monday, February 15, 2010

Retrieve Grails Domain Errors From the 'messageSource'

As you may know, we can add certain constraints to our Grails domains to ensure the integrity of our priceless data.

Suppose we have a simple domain:


class AppUser{
String login
String password

static constraints = {
login unique:'true', blank:'false'
password blank:'false'
}
}


Here, we have set up our domain so that our login not only needs to be a non-blank value but it also has to be unique.

When saving a new user in something like a service, we can take the following approach:


if (!user.save()) {
log.error "Failed to Save User"
user.errors.allErrors.each {log.error it.defaultMessage}
}


So lets just concentrate on the unique constraint. If that constraint is broken, the 'user.save()' will evaluate to false. We then query the domain for its errors using 'users.errors.allErrors' and iterate over each error using the each{} method that is provided by Groovy to all collections. (We can throw an exception or take another action after this if you wish :) )

OK.... so after checking the log, I was a little confused as my 'log.error it.defaultMessage' was printing something cryptic like:


Property [{0}] of class [{1}] with value [{2}] must be unique


Hmmm.... not very nice... this is where the Grails messageSource bean comes into play. The messageSource can be used to translate our cryptic error message to something more digestible..



class MyService{
def messageSource // inject the messageSource

def saveUser(AppUser user) {
if (!user.save()) {
log.error "Failed to Save User"
user.errors.allErrors.each {log.error messageSource.getMessage(it, null)}
}
}
}

Now our error message will be read from messages.properties in grails-app/i18n and look something like:

Property [login] of class [AppUser] with value [john] must be unique

Huzzah.... readable messages.... (Note that in the call to 'messageSource.getMessage(it, null)', the null refers to the locale. We won't cover that here...)

If we check our messages.properties file mentioned previous we can see where this default message came from:


default.not.unique.message=Property [{0}] of class [{1}] with value [{2}] must be unique


The beauty is... we can provide our own message to make the error message even cleaner for the user... we just need to add the following to message.properties :


com.company.project.AppUser.login.unique=Login must be a unique value!


Basically we use the path to our class, followed by the field then followed by the constraint.

OK, so one final thing was bugging me.... do I need to include code in every service method to iterate over the errors and get the correct messages when validation fails, this wouldn't be very dry and rather .... boring....

Hmmm how could I solve this one???? I know... metaprogramming.... c'mon.. stay with me!!!

The aim: add a method to all my domain classes that would 'collect' all the messages for each error and return them in a readable format. The best place to do this is probably the BootStrap class. So lets give it a try:


class BootStrap{
def messageSource //our old friend
def grailsApplication//even older friend :)

def init = {servletContext ->
grailsApplication.domainClasses.each {domainClass ->//iterate over the domainClasses
if (domainClass.clazz.name.contains("com.company.project")) {//only add it to the domains in my plugin

domainClass.metaClass.retrieveErrors = {
def errorString = delegate?.errors?.allErrors?.collect{messageSource.getMessage(it,null)}?.join(' \n')

return errorString
}
}
}
}//end of init
}



Phew... Taking it one step at a time, we first iterate over the applications domain classes. A check is performed to ensure that we only add the method to our own domains (optional) and the method retrieveErrors() is added to the meta class of each of our domains. We then have a fancy (or egotistical) one liner that collects all the errors of the delegate domain, evaluates all their error messages into a List and joins them with a line break. Let's break up the one liner some what:


def init = {servletContext ->
grailsApplication.domainClasses.each {domainClass ->//iterate over the domainClasses
if (domainClass.clazz.name.contains("com.company.project")) {//only add it to the domains in my plugin

domainClass.metaClass.retrieveErrors = {
def list = delegate?.errors?.allErrors?.collect{messageSource.getMessage(it,null)}
return list?.join('\n')
}
}
}
}//end of init



In the end we have replaced :

if (!user.save()) {
log.error "Failed to Save User"
user.errors.allErrors.each {log.error messageSource.getMessage(it, null)}

with:


if (!user.save()) log.error "Failed to save user : ${user.retrieveErrors())}"


One final tidbit.... If you find yourself needing the messageSource outside of a Grails artefact... you can try the following:


MessageSource messageSource = ApplicationHolder.application.mainContext.getBean('messageSource')


The import requred is 'import org.springframework.context.MessageSource'

Hope this helps!

Info:

http://grails.org/doc/latest/ref/Constraints/validator.html
http://grails.org/doc/latest/ref/Constraints/Usage.html