The inner optimization is trying to find the maximum of the
log-likelihood function wrt the random effects parameters conditioned on
the fixed values of the other parameters. There is some numerical
problem at this point. If you post your code I can take a look at it,
and post a note on how to diagnose problems in this case.
Dave