[ADMB Users] mceval output inconsistent with mcmc display

Fri Jun 24 02:39:06 PDT 2011

Solved it. Turns out it was related to the windows-linux inconsistencies
after all. At some point on my journey to the full 600+ parameter model
the linux systems told me the hessian matrix was singular, however the
windows systems would carry merrily on with MCMC. Like a fool with a
deadline I believed the implementation I wanted to (ie windows), but in
hindsight this should have been a neon sign. I eventually realised some
of my spatio-temporal anomaly parameters were unidentifiable and
switched them off / reparameterised. The linux system then gave me
sensible posteriors. 

You appear to have a bug in the windows implementations - they are
failing to identify singular hessians, and are nevertheless executing a
highly suspicious matrix decomposition required for the MCMC jump step.
In fact the covariance matrix actually had a negative eigenvalue
(indicating the "ML" estimate was in fact sitting at a saddle point) but
this may have just been due to roundoff error (was reading from the .cov
file so 4 decimal point precision which for this purpose is way
insufficient). Still not sure why this should produce the behaviour
listed below, but the windows MCMC for that model, which I now know to
be unidentifiable, was definitely (and not surprisingly) producing
rubbish. 

I wouldn't have thought the windows and linux systems use different
libraries / algorithms for matrix ops? Note this is not a 32-64bit
thing, as both 32bit and 64bit windows systems showed the same
behaviour. 

Dave I'll email you the code and inputs that reproduce this bug off
list.

Thanks to my colleague George Leigh for helping me figure this one out
(in fact for doing much of the figuring!)

-alex 

-----Original Message-----
From: dave fournier [mailto:davef at otter-rsch.com] 
Sent: Friday, 10 June 2011 2:43 PM
To: Campbell, Alex
Cc: users at admb-project.org
Subject: Re: [ADMB Users] mceval output inconsistent with mcmc display

On 11-06-09 09:23 PM, Campbell, Alex wrote:

Still not enough information to understand completely.
However do you have real priors on all the parameters.
I recall fitting a model where the posterior was improper.

That means the integral over all the parameters was infinity.
This manifested itself in the parameters from the mcmc looking OK for a
while and then just wandering off to what would like like junk values.
the problem is that if you have a set of "reasonable" values with
proabability eps and a huge set of stupid values with probability 1-eps
if you run the mcmc long enough it will spend most of its time lost in
the stupid values.

> No, I'm concerned that the displayed likelihood is not consistent with

> the likelihood corresponding to the parameters obtained via mceval at 
> that point (at that iteration).
>
> Here's some output. Beginning of mcmc:
>
> Initial seed value -36519
> 785.580219414446 785.580219414446
>   mcmc sim 1  acceptance rate 0 0
> 785.580219414446 785.580219414446
>   mcmc sim 201  acceptance rate 0 0
> decreasing step size -2049.27939913456
> 750.625386694123 750.625386694123
>   mcmc sim 401  acceptance rate 0.0773067331670823 0.155
> 739.336552314425 739.336552314425
>   mcmc sim 601  acceptance rate 0.114808652246256 0.19
>
> Last few hundred iterations:
>
> 602.449537531548 602.449537531548
>   mcmc sim 999401  acceptance rate 0.036780031238712 0.03
> 602.366020956481 602.366020956481
>   mcmc sim 999601  acceptance rate 0.0367746730945647 0.01
> 603.32884634525 603.32884634525
>   mcmc sim 999801  acceptance rate 0.0367683168950621 0.005
>
> And not much variation in between - lowest likelihood around 590.
>
> But the likelihood for parameters read from psv by mceval is terrible,

> around -600, at least after the first few hundred iterations.
>
> The trace of primary parameters shows them moving from a sensible 
> region very quickly to a rubbish region.
>
>
> -----Original Message-----
> From: dave fournier [mailto:davef at otter-rsch.com]
> Sent: Friday, 10 June 2011 1:08 PM
> To: Campbell, Alex
> Subject: Re: [ADMB Users] mceval output inconsistent with mcmc display
>
> On 11-06-09 07:34 PM, Campbell, Alex wrote:
>
>
> So if I understand this you think the value should be closer to the 
> mode.
> Intuition about this is wrong.  In high dimensions you never get 
> anywhere near the mode.
> Think of a standard n dimensional multivariate normal. You can easily 
> see that the pdf for the distance r from 0 is prop. to 
> r^{n-1}*exp(-r^2/2}.
> Do a quick calc to see what the probability is that you will get to 
> within distance 1 of 0 when the dimension is 100. You may be
surprised.
>
>
>
>> Sorry yes it is the log likelihood. I was unclear.
>>
>> Mode estimate: approx 750 log likelihood. MCMC display output: ranges

>> from 600 to 750 over one million iterations. Parameters read from psv

>> file using mceval: around -600 after the first few hundred
iterations.
>>
>> Thanks Ian that's exactly the sort of investigative tool I think I
> need.
>> -alex
>>
>> -----Original Message-----
>> From: users-bounces at admb-project.org
>> [mailto:users-bounces at admb-project.org] On Behalf Of dave fournier
>> Sent: Friday, 10 June 2011 8:41 AM
>> To: users at admb-project.org
>> Subject: Re: [ADMB Users] mceval output inconsistent with mcmc 
>> display
>>
>> Probably better to make a little example first.
>> for example
>> DATA_SECTION
>> PARAMETER_SECTION
>>      init_vector x(1,10)
>>      sdreport_vector rx(1,10);
>>      objective_function_value f
>> PROCEDURE_SECTION
>>      f=0.5*norm2(x);
>>      rx=x;
>>      if (mceval_phase())
>>      {
>>        cout<<   f<<   ""<<   x<<   endl;
>>      }
>>
>> then running it with
>>
>>     ./small -mcmc 100000 -mcsave 1
>>
>> and the screen shows
>>
>> mcmc sim 1  acceptance rate 0 0
>> -4.04084 -4.04084
>>     mcmc sim 201  acceptance rate 0.169154 0.17
>> -2.76404 -2.76404
>>     mcmc sim 401  acceptance rate 0.187032 0.205
>> -5.52107 -5.52107
>>     mcmc sim 601  acceptance rate 0.189684 0.195
>> -6.00142 -6.00142
>>     mcmc sim 801  acceptance rate 0.187266 0.18
>> -4.85913 -4.85913
>>     mcmc sim 1001  acceptance rate 0.182817 0.165
>> -1.51622 -1.51622
>>     mcmc sim 1201  acceptance rate 0.181515 0.175
>> -3.43209 -3.43209
>>
>> So this is the log-likelihood *NOT* minus the log-likelihood. Also it

>> is every 200'th value. (Just printed out for amusement).
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ~
>>
>> _______________________________________________
>> Users mailing list
>> Users at admb-project.org
>> http://lists.admb-project.org/mailman/listinfo/users
>>
>>
>> ********************************DISCLAIMER***************************
>> * The information contained in the above e-mail message or messages 
>> (which includes any attachments) is confidential and may be legally 
>> privileged.  It is intended only for the use of the person or entity 
>> to which it is addressed.  If you are not the addressee any form of 
>> disclosure, copying, modification, distribution or any action taken 
>> or omitted in reliance on the information is unauthorised.  Opinions 
>> contained in the message(s) do not necessarily reflect the opinions 
>> of the Queensland Government and its authorities.  If you received 
>> this communication in error, please notify the sender immediately and

>> delete it from your computer system network.
>>
>>
>>
>
>
> ********************************DISCLAIMER****************************
> The information contained in the above e-mail message or messages 
> (which includes any attachments) is confidential and may be legally 
> privileged.  It is intended only for the use of the person or entity 
> to which it is addressed.  If you are not the addressee any form of 
> disclosure, copying, modification, distribution or any action taken or

> omitted in reliance on the information is unauthorised.  Opinions 
> contained in the message(s) do not necessarily reflect the opinions of

> the Queensland Government and its authorities.  If you received this 
> communication in error, please notify the sender immediately and 
> delete it from your computer system network.
>
>
>

********************************DISCLAIMER****************************
The information contained in the above e-mail message or messages 
(which includes any attachments) is confidential and may be legally 
privileged.  It is intended only for the use of the person or entity 
to which it is addressed.  If you are not the addressee any form of 
disclosure, copying, modification, distribution or any action taken 
or omitted in reliance on the information is unauthorised.  Opinions 
contained in the message(s) do not necessarily reflect the opinions 
of the Queensland Government and its authorities.  If you received 
this communication in error, please notify the sender immediately 
and delete it from your computer system network.