The calculation for random effects is explained in the ADMB-RE manual, although the presentation is a bit unclear at least to me and I wrote the code. Take a look at it and see if it helps.