[Developers] admb threading

[John Sibert]

Matthew,

Putting low-level locks on variable objects (ie dvariable, 
df1b2variable) is equivalent to blocking every floating point operation 
in the model, and is guaranteed to slow things down a bit. Blocking the 
gradient structure would have the same effect. In both cases, I can 
imagine problems with the order in which the reverse calculations in 
gradcalc() are executed. This approach leaves gradcalc() as a single 
thread. Since the reverse computation typically uses 2-5 times as many 
floating point operations as the forward computation, it would be a good 
idea to find a way to do these in parallel.

Duplicating the gradient structures and making deep copies of objects 
involved in derivative computations seems at first to be a bit 
inefficient. Perhaps, but memory is cheap and creating thread-specific 
gradient structures ensures that the derivatives can be correctly computed.

I'm not saying that your approach will absolutely not work. It might 
work (in the sense that you can make the derivatives correct), but 
performance benefits will be difficult to measure and will be 
application specific.

Here is a cautionary tale. About 15 years ago, I implemented pthread in 
an easily parallel section of model for which I had written my own 
derivative code. The only way it would work was to make separate data 
structures for each thread in which the derivative information was 
stored. The derivatives were correct, and it certainly decreased the 
amount of time spent in that particular section of the code. Alas, the 
model as a whole only performed slightly better because a different 
bottleneck came to the top. So you need thread-specific gradient 
information, and there is no guarantee that things will run faster. The 
performance benefit is application specific.

So for now, Johnoel and I will try to implement Dave's approach. We will 
be looking for operations that are inherently parallelizable (Hessian 
computations, separable functions, ... ), civilizing
Dave's protoype code to make it a bit safer, extending the concept to 
higher dimensional objects, and exploring was to easily implement 
parallel sections of user models (such as those which use the funnel).

Thanks for sharing your thoughts and questions. It is one of the 
benefits of open source.

Cheers,
John

John Sibert
Emeritus Researcher, SOEST
University of Hawaii at Manoa

Visit the ADMB project http://admb-project.org/

On 12/04/2012 03:46 PM, Matthew Supernaw wrote:
> I misinterpreted your joke, I apologize.  How about we agree to be polite and continue?
>
> Would a global mutex lock on the gradient_structure work?
>
> If it would work, what is more costly copying data into a master/slave paradigm or having an operational bottle neck at the gradient_structure?
>
> On Dec 4, 2012, at 4:46 PM, dave fournier wrote:
>
>> On 12-12-04 01:15 PM, Matthew Supernaw wrote:
>>> Which book,  "Concurrent Programming for Procedural Programmers"?
>> Its usually called a joke.  I was referring to the code in my example.
>>
>>> Anyway,  lets step through the logic of:
>>>
>>> prevariable& operator+(const prevariable& v1, const prevariable& v2)
>>>
>>>
>>> you have a call to:
>>>
>>> gradient_structure::GRAD_STACK1->set_gradient_stack4(void(*),double*,double*double*)
>> Go ahead. Do what you want.
>>> and then:
>>>
>>> default_evaluation4()
>>>
>>> Why can't there be a mutex lock on gradient_structure?
>>>
>>>
>>>
>>>
>>>
>>> On Dec 4, 2012, at 3:00 PM, developers-request at admb-project.org wrote:
>>>
>>>> Send Developers mailing list submissions to
>>>> 	developers at admb-project.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>> 	http://lists.admb-project.org/mailman/listinfo/developers
>>>> or, via email, send a message with subject or body 'help' to
>>>> 	developers-request at admb-project.org
>>>>
>>>> You can reach the person managing the list at
>>>> 	developers-owner at admb-project.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of Developers digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>>    1. Re: admb threading (dave fournier)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Tue, 04 Dec 2012 11:56:50 -0800
>>>> From: dave fournier <davef at otter-rsch.com>
>>>> To: developers at admb-project.org
>>>> Subject: Re: [Developers] admb threading
>>>> Message-ID: <50BE5582.3000002 at otter-rsch.com>
>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>
>>>> Read the book!
>>>>
>>>> A way to synchronize is in my example referred to earlier. Of course
>>>> that is
>>>> just for dvariables. For df1b2 variables there is a lot more structure,
>>>> but the ideas
>>>> would be the same.
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> Developers mailing list
>>>> Developers at admb-project.org
>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>
>>>>
>>>> End of Developers Digest, Vol 46, Issue 10
>>>> ******************************************
>>> _______________________________________________
>>> Developers mailing list
>>> Developers at admb-project.org
>>> http://lists.admb-project.org/mailman/listinfo/developers
> _______________________________________________
> Developers mailing list
> Developers at admb-project.org
> http://lists.admb-project.org/mailman/listinfo/developers
>