[Developers] OpenMPI
dave fournier
davef at otter-rsch.com
Tue Jan 10 12:45:34 PST 2012
On 12-01-10 12:15 PM, Hans J. Skaug wrote:
Its a low end AMD triple core. With the parallelized Hess stuff I did
1 slave worked better than 0 but 2 were worse than 1.
We should try a high end mult-processor system.
I view Derek's current version as a proof of concept. One thing to check
is whether we can use non blocking send/receive between the masters and
the slaves.
(assuming it is not used now.) Also I think that one can use a broadcast
send to slaves
when appropriate. There are also various profiling tools.
You can look at openmpi docs to see how it is installed. There is not
much to it.
Proably your linux dist has it already.
It uses gcc with appropriate extra flags. these can be automatically used by
using mpicc which is just a shell to call gcc with the right flags
that way if open mpi changes the flags it just works for you.
what I showed was how to modify the make call for use with mpi
CC=gcc CXX=g++ CXXFLAGS="-m64 $(shell mpicc --showme:compile)
this uses g++ but
$(shell mpicc --showme:compile)
produces a string with the right flags for open mpi.
> Dave: How many cores do you have on your machine?
>
> Derek: I have never used open MPI before. Can you give
> me summary of what I need to do if I want to
> compile your code.
> - How do I know if open mpi is installed on my linux box?
> - Is the make file in
>
> svn co svn+ssh://www.admb-project.org/branches/parallel admb-mpi
>
> set up to compile with MPI? (Dave's previous email suggests no).
>
> Hans
>
>> -----Original Message-----
>> From: dave fournier [mailto:davef at otter-rsch.com]
>> Sent: Tuesday, January 10, 2012 7:27 PM
>> To: Derek Seiple
>> Cc: Hans J. Skaug; 'Arni Magnusson'; 'anders at nielsensweb.org'
>> Subject: Re: OpenMPI
>>
>> On 12-01-09 05:42 PM, Derek Seiple wrote:
>>
>> Hi,
>>
>> Using time ./netsted4 -nohess I get
>>
>> real 0m14.405s
>> user 0m14.265s
>> sys 0m0.132s
>>
>> Using time ./nested4 -nohess -master I get
>>
>>
>> real 0m11.404s
>> user 0m11.160s
>> sys 0m0.223s
>>
>> This is for the slow safe debug version
>> As I said hessian calculations are broke anyway and that is
>> a different issue so you should try the -nohess option.
> >from what I have it looks like your mpi stuff speeds it up,
>> although not by a huge amount.
>>
>> I think it would be better to extend your work to date to the
>> dense hessian and aparse hessian cases rather than having
>> competing implementations. One thing. Whay cant you use the
>> existing separable_calls_counter and num_separable calls instead of
>> adding a new one. If that is OK we could add a global_num_separable_calls
>> for when we really need them all.
>>
>> Dave
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> There are some other examples in the mpi_tests folder. One that takes
>>> a while to run for me is the nested4 example. When I would run it
>>> normally it would take about 40 seconds or so, but with a slave and
>>> master it takes over a minute.
>>>
>>> Derek
>>>
>>> On Mon, Jan 9, 2012 at 5:31 PM, dave fournier<davef at otter-rsch.com>
>> wrote:
>>>> On 12-01-09 07:51 AM, Derek Seiple wrote:
>>>>
>>>> Hi,
>>>>
>>>> this seems to run OK for the estimation. Do you have a large example?
>>>>
>>>> The Hessian calculations blow up I think because something like
>>>> separable_calls_counter is
>>>> not getting initialized.
>>>>
>>>> Dave
>>>>
>>>>
>>>>
>>>>
>>>>> Hi Dave,
>>>>>
>>>>> Probably the easiest way is for you to checkout the code using the
>>>>> following:
>>>>>
>>>>> svn co svn+ssh://www.admb-project.org/branches/parallel
>>>>> <your_destination_folder>
>>>>>
>>>>> That is the most up-to-date code. If that doesn't work for you i can
>>>>> try to zip up the source and email it to you...
>>>>>
>>>>> I compile the source with the GNUmakefile but that probably wont work
>>>>> for you. I think GNUmakefile.64bit might be the one you used. The
>>>>> build process isn't that smooth yet in the parallel branch.
>>>>>
>>>>> I also add these 2 lines to my .bashrc file
>>>>> export
>> ADMB_HOME=<path_to_where_you_downloaded_the_source>/build/unix/
>> unix
>>>>> export PATH=$ADMB_HOME/bin:$PATH
>>>>>
>>>>> Then you can go to mpi_tests/orange_mpi_test and then just type make
>>>>> or make debug to compile the example.
>>>>> After that
>>>>> ./orange it runs it as normal
>>>>> ./orange -master runs it with one master and one slave
>>>>> ./orange -master -nslaves n runs it with one master and n slaves
>>>>>
>>>>> Derek
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jan 8, 2012 at 12:36 PM, dave fournier<davef at otter-rsch.com>
>>>>> wrote:
>>>>>> On 11-11-18 06:04 AM, Derek Seiple wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> If you want me to look at this you should send me all the files necessary
>>>>>> to
>>>>>> run it
>>>>>> or tell me how to get them. I don;t want to spend a lot of time and then
>>>>>> find out
>>>>>> I'm not using the latest source.
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi Dave,
>>>>>>>
>>>>>>> I have the separable case done. The only problem is that it isn't
>>>>>>> really any faster. Some models, even a little slower. Can you look at
>>>>>>> my code and see if you see any obvious slow downs.
>>>>>>>
>>>>>>> I suspect the code in df1b2lp9.cpp is probably the cause of the slow
>>>>>>> down, but I don't know what to do with it right now.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Derek
>>>>>>>
>>>>>>> On Sun, Nov 13, 2011 at 7:25 PM, dave fournier<davef at otter-
>> rsch.com>
>>>>>>> wrote:
>>>>>>>> On 11-11-11 09:05 AM, Derek Seiple wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Actually that is just the simplest case.
>>>>>>>>
>>>>>>>> But anyway just to clean up the code. I think that anything to do with
>>>>>>>> MPI should be inside a
>>>>>>>>
>>>>>>>>
>>>>>>>> #if defined(USE_ADMPI)
>>>>>>>>
>>>>>>>> #endif
>>>>>>>>
>>>>>>>> so that we can still compile the code without any MPI stuff.
>>>>>>>> I try to do it that way but things keep sneaking out.
>>>>>>>>
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>> Ok.
>>>>>>>>>
>>>>>>>>> I am finishing up the RE hessian stuff now. Once that is done, then I
>>>>>>>>> think the separable stuff will be done.
>>>>>>>>>
>>>>>>>>> Derek
>>>>>>>>>
>>>>>>>>> On Fri, Nov 11, 2011 at 12:01 PM, dave fournier<davef at otter-
>> rsch.com>
>>>>>>>>> wrote:
>>>>>>>>>> On 11-11-11 08:56 AM, dseiple84 at gmail.com wrote:
>>>>>>>>>>
>>>>>>>>>> I don;t see how it could be. It was only when creating the MPI code
>>>>>>>>>> that
>>>>>>>>>> the
>>>>>>>>>> call to sd_routine got left out. So it was never part of the main
>>>>>>>>>> distribution.
>>>>>>>>>> However once I fix that std files do get created when I run a simple
>>>>>>>>>> example
>>>>>>>>>> using
>>>>>>>>>> your admb-mpi code.
>>>>>>>>>>
>>>>>>>>>>> Is this related to one of the earlier emails about std files from
>>>>>>>>>>> version
>>>>>>>>>>> 9 to version 10?
>>>>>>>>>>>
>>>>>>>>>>> On , dave fournier<davef at otter-rsch.com> wrote:
>>>>>>>>>>>> On 11-11-08 11:39 AM, Derek Seiple wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There was an error in this whihc cuased std files not to get
>>>>>>>>>>>> printed
>>>>>>>>>>>> out
>>>>>>>>>>>> sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>> It is only in the open mpi branch so far as I know.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Dave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Never mind. I see now that the RE are set during the Newton
>> Raphson
>>>>>>>>>>>> inside evaluate_function(uhat,pfmin)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Nov 8, 2011 at 9:18 AM, Derek
>> Seipledseiple84 at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Dave,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I now haw the minimizer converging for multiple slaves. One
>> thing I
>>>>>>>>>>>> haven't figured out is where the random effects values are
>> stored
>>>>>>>>>>>> and
>>>>>>>>>>>>
>>>>>>>>>>>> where they get set to varsptr pr printing in the par file.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I know that the vector y in the laplace is comprised of y=(x,u)
>>>>>>>>>>>> that
>>>>>>>>>>>>
>>>>>>>>>>>> is the variable and the random effects. However just updating
>> the u
>>>>>>>>>>>> protion of y in the master doesn't get the values in to the par
>>>>>>>>>>>> file.
>>>>>>>>>>>>
>>>>>>>>>>>> For example this is the par file I get:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> # Number of parameters = 5 Objective function value = 94.8143
>>>>>>>>>>>>
>>>>>>>>>>>> Maximum gradient component = 5.56233e-05
>>>>>>>>>>>>
>>>>>>>>>>>> # beta:
>>>>>>>>>>>>
>>>>>>>>>>>> 0.0531627 1.90635 -7.92692
>>>>>>>>>>>>
>>>>>>>>>>>> # log_sigma:
>>>>>>>>>>>>
>>>>>>>>>>>> 2.05962297763
>>>>>>>>>>>>
>>>>>>>>>>>> # log_sigma_u:
>>>>>>>>>>>>
>>>>>>>>>>>> 3.45462211398
>>>>>>>>>>>>
>>>>>>>>>>>> # u:
>>>>>>>>>>>>
>>>>>>>>>>>> -29.5621226547 31.7279669314 -37.1934482041 0.00000000000
>>>>>>>>>>>> 0.00000000000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So the random effects are stored somewhere (besides y). I
>> need to
>>>>>>>>>>>> find
>>>>>>>>>>>>
>>>>>>>>>>>> out where so I can send the RE from the slaves to the master.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Nov 4, 2011 at 5:17 PM, dave fournierdavef at otter-
>> rsch.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 11-11-04 01:43 PM, Derek Seiple wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That seems right. So all that needs to be done is to take
>>>>>>>>>>>>
>>>>>>>>>>>> the f's from the slaves and the g from the slsves and
>>>>>>>>>>>>
>>>>>>>>>>>> add them to the f and the g from the master.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Keep in mind that this is for the simple block diagonal case.
>>>>>>>>>>>>
>>>>>>>>>>>> because the same random effects do not occur in two
>> processes.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So the dvector uhat that get calculated by
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> uhat=get_uhat_quasi_newton_block_diagonal(x,pfmin);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> is the inner optimization. And then
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> for(int ii=1;ii
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> // test newton raphson
>>>>>>>>>>>>
>>>>>>>>>>>> //Hess.initialize();
>>>>>>>>>>>>
>>>>>>>>>>>> int check=initial_params::stddev_scale(scale,uhat);
>>>>>>>>>>>>
>>>>>>>>>>>> check=initial_params::stddev_curvscale(curv,uhat);
>>>>>>>>>>>>
>>>>>>>>>>>> max_separable_g=0.0;
>>>>>>>>>>>>
>>>>>>>>>>>> pmin->inner_opt_flag=1;
>>>>>>>>>>>>
>>>>>>>>>>>> step=get_newton_raphson_info_block_diagonal(pfmin);
>>>>>>>>>>>>
>>>>>>>>>>>> cout
>>>>>>>>>>>> cout
>>>>>>>>>>>> uhat+=step;
>>>>>>>>>>>>
>>>>>>>>>>>> evaluate_function(uhat,pfmin);
>>>>>>>>>>>>
>>>>>>>>>>>> pmin->inner_opt_flag=0;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> is the Newton-Raphson part that adjusts uhat to make
>>>>>>>>>>>>
>>>>>>>>>>>> F_u(x,uhat(x))=e
>>>>>>>>>>>>
>>>>>>>>>>>> even smaller?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The above snippets get called within
>>>>>>>>>>>>
>>>>>>>>>>>> g=(*lapprox)(x,f,this);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The resultant g is the gradient of the function
>>>>>>>>>>>>
>>>>>>>>>>>> L(x) = f(x,uhat(x)).
>>>>>>>>>>>>
>>>>>>>>>>>> Correct?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This g and corresponding f and x then get fed to the outer
>>>>>>>>>>>> maximization.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Nov 4, 2011 at 2:20 PM, dave fournierdavef at otter-
>> rsch.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 11-11-04 10:57 AM, Derek Seiple wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No the inner optimization(s) use the function minimizer to do
>>>>>>>>>>>>
>>>>>>>>>>>> initial optimization.Tbhe problme was that qusi newton can not
>> in
>>>>>>>>>>>> general
>>>>>>>>>>>>
>>>>>>>>>>>> get the gradient small enough for the total function to be
>> smooth
>>>>>>>>>>>> enough.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Recall I told you that the chain rule is used in
>>>>>>>>>>>>
>>>>>>>>>>>> differentiating
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> F_u(x,uhdat(x))=0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> to solve for uhat'(x) = -inv(F_uu) * F_xu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But actually F_u(x,uhdat(x))=e
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> where e is (hopefully) small.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> To make e small the newton raphson is used after the quasi-
>> newton.
>>>>>>>>>>>> For the block diagonal hessian each block has its own inner
>>>>>>>>>>>> minimization
>>>>>>>>>>>>
>>>>>>>>>>>> and newton-raphson.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> After this is done then the laplace approximation is carried out.
>>>>>>>>>>>> So
>>>>>>>>>>>> it
>>>>>>>>>>>>
>>>>>>>>>>>> all
>>>>>>>>>>>>
>>>>>>>>>>>> looks like this,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 1.) inner quasi newton (uses dvariables)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2.)Newton raphson 1 using df1b2variable to calculate hessian H
>>>>>>>>>>>>
>>>>>>>>>>>> update is inv(H)*g
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 3.)Newton raphson 2 using df1b2variable to calculate hessian H
>>>>>>>>>>>>
>>>>>>>>>>>> update is inv(H)*g
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> calculate H using df1b2variables and then calucate some
>> function of
>>>>>>>>>>>> H
>>>>>>>>>>>>
>>>>>>>>>>>> using dvariables this is essentially log(det(H)) for the Laplace
>>>>>>>>>>>>
>>>>>>>>>>>> approximation
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Now do reverse AD on log(det(H)) to get derviatives back to
>> where H
>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>> calculated.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Paste that into the final reverse for df1b2variables
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> You should just make a tiny toy program and follow this through
>>>>>>>>>>>> with
>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> debugger.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> So if I understand how it works, the inner maximizations happen
>> in
>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> laplace_aprox correct? Then the out maximization is done with
>> the
>>>>>>>>>>>> function_minimizer?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If this is the case then how should the following code be
>> executed?
>>>>>>>>>>>>
>>>>>>>>>>>> This is OK at least for the first go. Later one might want to
>>>>>>>>>>>>
>>>>>>>>>>>> parallelize
>>>>>>>>>>>>
>>>>>>>>>>>> the outer one as well but the jobs are separate.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> if (negdirections==0)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> while (fmc.ireturn>=0)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> fmc.fmin(f,x,g); //
>>>>>>>>>>>> if (fmc.ireturn>0)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> if (ifn_trap)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> if (ifn_trap==fmc.ifn&& itn_trap == fmc.itn)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> cout
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> g=(*lapprox)(x,f,this); //
>>>>>>>>>>>> if (bad_step_flag==1)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> g=1.e+4;
>>>>>>>>>>>>
>>>>>>>>>>>> f=2.*fmc.fbest;
>>>>>>>>>>>>
>>>>>>>>>>>> bad_step_flag=0;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> if (lapprox->init_switch==0)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> if (f
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> lapprox->ubest=lapprox->uhat;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 3, 2011 at 7:40 PM, dave fournierdavef at otter-
>> rsch.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 11-11-03 02:52 PM, Derek Seiple wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There is an inner maximization (actually a bunch of them)
>>>>>>>>>>>>
>>>>>>>>>>>> and an outer maximization. I assume that the failure is
>>>>>>>>>>>>
>>>>>>>>>>>> in the outer maximization. all you need to do is to compare
>>>>>>>>>>>>
>>>>>>>>>>>> the gradient and the function value that is going to the
>>>>>>>>>>>>
>>>>>>>>>>>> function minimizier. One or both of them must have changed.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It is a waste of time worrying about what goes on inside the
>>>>>>>>>>>>
>>>>>>>>>>>> function minimizer itself. If yoiu feed it the correct function
>>>>>>>>>>>> value
>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>>
>>>>>>>>>>>> gradient
>>>>>>>>>>>>
>>>>>>>>>>>> it will work.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> But remember that you should only be running one outer
>> maximization
>>>>>>>>>>>> in the master. I'm worried about what the slaves are doing. YOu
>>>>>>>>>>>> should
>>>>>>>>>>>>
>>>>>>>>>>>> make
>>>>>>>>>>>>
>>>>>>>>>>>> sure they only do what you want them to do.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> fmm::fmin in newfmin.cpp is tough to read because of al the
>> goto's.
>>>>>>>>>>>> But one thing that is common amongst all the examples I've run
>> with
>>>>>>>>>>>> 2
>>>>>>>>>>>>
>>>>>>>>>>>> or more slaves ends up with
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Function minimizer: Step size too small -- ialph=1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It is difficult to track down what trips this; even with the
>>>>>>>>>>>> debugger.
>>>>>>>>>>>>
>>>>>>>>>>>> Do you know what types of things cause that step size message.
>>>>>>>>>>>>
>>>>>>>>>>>> Particularly anything related to separable models.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Figuring this out is important to understanding why the
>> minimizer
>>>>>>>>>>>> doesn't converge, because right after the step size message the
>>>>>>>>>>>>
>>>>>>>>>>>> function value blows up.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Interestingly this doesn't happen with only one slave. The
>>>>>>>>>>>> minimizer
>>>>>>>>>>>>
>>>>>>>>>>>> converges.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 3, 2011 at 10:26 AM, Derek
>> Seipledseiple84 at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Alternatively, did you compile with either
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> GNUMakefile.js or GNUMakefile.64bit?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> They should all be in one make file but I haven't done that.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Nov 3, 2011 at 10:24 AM, Derek
>> Seipledseiple84 at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> What is the path you have set to ADMB_HOME? On my machine
>> I have
>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> following in bashrc
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> export ADMB_HOME=/home/dseiple/admb-
>> mpi/build/unix/unix
>>>>>>>>>>>> export PATH=$ADMB_HOME/bin:$PATH
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If you look in orange.cpp there should be this at the top
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> extern ad_separable_manager * separable_manager;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If it isn't there then you probably tried to compile with another
>>>>>>>>>>>>
>>>>>>>>>>>> version of ADMB. Or maybe you need to make clean in order for
>> the
>>>>>>>>>>>> modified sed script to make it into the build directory?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 2, 2011 at 5:06 PM, dave fournierdavef at otter-
>> rsch.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 11-11-02 12:13 PM, Derek Seiple wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> your code won't compile for me any more. I get
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp:12:8: error: 'ad_separable_manager' does not name
>> a type
>>>>>>>>>>>> orange.cpp: In member function 'virtual void
>>>>>>>>>>>>
>>>>>>>>>>>> model_parameters::userfunction()':
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp:41:20: error: 'sb' was not declared in this scope
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp:41:31: error: 'separable_bounds' was not declared in
>>>>>>>>>>>>
>>>>>>>>>>>> this
>>>>>>>>>>>>
>>>>>>>>>>>> scope
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp: In function 'int main(int, char**)':
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp:106:7: error: 'dp' was not declared in this scope
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp: In member function 'virtual void
>>>>>>>>>>>>
>>>>>>>>>>>> df1b2_parameters::user_function()':
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp:161:20: error: 'sb' was not declared in this scope
>>>>>>>>>>>>
>>>>>>>>>>>> orange.cpp:161:31: error: 'separable_bounds' was not declared
>> in
>>>>>>>>>>>> this
>>>>>>>>>>>>
>>>>>>>>>>>> scope
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Error: could not create orange.o
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I added another example and changed some code in
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>> laplace_approximation_calculator::get_uhat_quasi_newton_block_diagonal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It appears that things converge in both models when I only have
>> 1
>>>>>>>>>>>> slave, but go to hell when I have 2 slaves.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Nov 1, 2011 at 4:05 PM, Derek
>> Seipledseiple84 at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Dave,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Can you take a look at the latest version of what I have in the
>>>>>>>>>>>>
>>>>>>>>>>>> repository?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> When I run the example model (or liver_gamma in the RE
>> examples
>>>>>>>>>>>> folder) the RE portion doesn't converge properly and I can't
>>>>>>>>>>>>
>>>>>>>>>>>> figure
>>>>>>>>>>>>
>>>>>>>>>>>> out why. I have compared values with the debugger and it
>> appears
>>>>>>>>>>>> that
>>>>>>>>>>>>
>>>>>>>>>>>> the evaluation of
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> fmc.fmin(f,x,g);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> g=(*lapprox)(x,f,this);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agree whether I use a slave or not, but the behavior is
>>>>>>>>>>>>
>>>>>>>>>>>> different.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Is there anything in the underlying code that I might be over
>>>>>>>>>>>>
>>>>>>>>>>>> looking?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> For example, I found that I had to add some code in gradcalc() to
>>>>>>>>>>>>
>>>>>>>>>>>> manually adjust
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> INDVAR_LIST from the grad_stack class to account for splitting
>>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> separable function.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Could there be something similar happening with RE stuff that I
>>>>>>>>>>>>
>>>>>>>>>>>> don't
>>>>>>>>>>>>
>>>>>>>>>>>> see yet? Something with the laplace? or the minimizer?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 16, 2011 at 3:12 PM, dave
>>>>>>>>>>>>
>>>>>>>>>>>> fournierdavef at otter-rsch.com>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 11-10-16 10:27 AM, Derek Seiple wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I started the program in phase 2 with and without a slave.
>>>>>>>>>>>>
>>>>>>>>>>>> you can see that both the function value and derivatives are
>>>>>>>>>>>>
>>>>>>>>>>>> different.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Dave,
>>>>>>>>>>>>
>>>>>>>>>>>> Nice work.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm running your orange example. Are there any special flags.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> No special flags. The way I run the example is just
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ./orange -master or
>>>>>>>>>>>>
>>>>>>>>>>>> ./orange -master -nslaves n
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> As near as I can tell in user function both master and slave
>>>>>>>>>>>>
>>>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>>> starting
>>>>>>>>>>>>
>>>>>>>>>>>> with
>>>>>>>>>>>>
>>>>>>>>>>>> i=1 in the for loop below
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> for(i=sb->indexmin();iindexmax();i++)
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>
>>>>>>>>>>>> ii = 0;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> When I started working on this I was having issues with the
>>>>>>>>>>>>
>>>>>>>>>>>> non-RE
>>>>>>>>>>>>
>>>>>>>>>>>> portion of the example. Since I was trying to get the RE stuff
>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> work
>>>>>>>>>>>>
>>>>>>>>>>>> I just made the sb->indexmin() call work so that both slave and
>>>>>>>>>>>>
>>>>>>>>>>>> master
>>>>>>>>>>>>
>>>>>>>>>>>> would do all the non-RE stuff and then once RE kicked in they
>>>>>>>>>>>>
>>>>>>>>>>>> would
>>>>>>>>>>>>
>>>>>>>>>>>> change so that slave and master would take different parts.
>> Now
>>>>>>>>>>>> that I
>>>>>>>>>>>>
>>>>>>>>>>>> understand how everything is working, I think I can change it
>>>>>>>>>>>>
>>>>>>>>>>>> (only
>>>>>>>>>>>>
>>>>>>>>>>>> need to change one line of code) so that the separable calls
>>>>>>>>>>>>
>>>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>>> split
>>>>>>>>>>>>
>>>>>>>>>>>> up all the time.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On a side note, have you been able to run the example on your
>>>>>>>>>>>>
>>>>>>>>>>>> machine?
>>>>>>>>>>>>
>>>>>>>>>>>> I know you said it crashed for you. I am still figuring out
>>>>>>>>>>>>
>>>>>>>>>>>> what
>>>>>>>>>>>>
>>>>>>>>>>>> some
>>>>>>>>>>>>
>>>>>>>>>>>> of the subtle issues are. When I run it with ./orange -master
>>>>>>>>>>>>
>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>
>>>>>>>>>>>> one slave) it converges and the par file has values pretty
>>>>>>>>>>>>
>>>>>>>>>>>> close
>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>> what you would get if you just ran ./orange (no slave), but the
>>>>>>>>>>>>
>>>>>>>>>>>> gradient values are a little off and it complains about the
>>>>>>>>>>>>
>>>>>>>>>>>> covariance
>>>>>>>>>>>>
>>>>>>>>>>>> matrix. When you run ./orange -master -nslaves 2 the
>>>>>>>>>>>>
>>>>>>>>>>>> convergence
>>>>>>>>>>>>
>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>> a
>>>>>>>>>>>>
>>>>>>>>>>>> little worse and the gradient values are even worse, that is
>>>>>>>>>>>>
>>>>>>>>>>>> not
>>>>>>>>>>>>
>>>>>>>>>>>> small
>>>>>>>>>>>>
>>>>>>>>>>>> and it complains about the hessian. (see the attached files)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This leads me to believe I am not handling the gradients
>>>>>>>>>>>>
>>>>>>>>>>>> properly.
>>>>>>>>>>>>
>>>>>>>>>>>> Late on Friday I realized I wasn't synchronizing the
>>>>>>>>>>>>
>>>>>>>>>>>> re_objective_function_value, but I haven't looked into that
>>>>>>>>>>>>
>>>>>>>>>>>> further
>>>>>>>>>>>>
>>>>>>>>>>>> yet. So I don't know if that will help things or if I am doing
>>>>>>>>>>>>
>>>>>>>>>>>> something wrong with the gradients.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Derek
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Oct 16, 2011 at 11:55 AM, dave
>>>>>>>>>>>>
>>>>>>>>>>>> fournierdavef at otter-rsch.com>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> BTW the best way to use the debugger ddd in linux is to put
>>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> following
>>>>>>>>>>>>
>>>>>>>>>>>> code at the
>>>>>>>>>>>>
>>>>>>>>>>>> beginning of the program
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> int ii = 0;
>>>>>>>>>>>>
>>>>>>>>>>>> char hostname[256];
>>>>>>>>>>>>
>>>>>>>>>>>> gethostname(hostname, sizeof(hostname));
>>>>>>>>>>>>
>>>>>>>>>>>> printf("PID %d on %s ready for attach\n", getpid(),
>>>>>>>>>>>>
>>>>>>>>>>>> hostname);
>>>>>>>>>>>>
>>>>>>>>>>>> fflush(stdout);
>>>>>>>>>>>>
>>>>>>>>>>>> while (0 == ii)
>>>>>>>>>>>>
>>>>>>>>>>>> sleep(5);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It will print out its PID and wait.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Then you can attatch to it with the debugger with
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ddd orange pid
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> do a backtrace and set ii=1
>>>>>>>>>>>>
>>>>>>>>>>>> and set a breakpoint where ever you want and continue.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>
More information about the Developers
mailing list