[Developers] admb threading
Matthew Supernaw
matthew.supernaw at noaa.gov
Tue Dec 4 10:39:13 PST 2012
So I just tried the threading example and it worked.
A couple of questions:
1. What are the consequences for modifying the copy constructor to make a deep copy(i'm sure there are many)?
2. Can "send_xxx_to_slave" be replaced with a deep copy function?
For instance:
df1b2variable DeepCopy(const df1b2variable &x){
return deep_copy_of_x;
}
class ADMBThread{
public:
df1b2variable x;
virtual void Run(){
do something to x….
}
};
int main(){
df1b2variable x = rand();
ADMBThread thread;
thread.x = DeepCopy(x);
thread.Start();
}
Thanks.
Matthew
On Dec 4, 2012, at 9:19 AM, Matthew Supernaw wrote:
>
> Dave,
>
> You are correct, I didn't review your example code. I'll take a look.
>
> Thanks for the architectural overview. It would be convenient if this information was recorded in software design document.
>
>
>
> "I guess you didn't bother wasting your time looking at my example for
> dvariables.
>
> The big problem with these objects is not some trivial locking in the
> constructor etc.
> It comes from the fact that the data structure which store all the
> information for
> multiple levels of reverse AD are not thread safe. If you are doing
> df1b2variable
> arithmetic each thread needs its own copy of these."
>
> On Dec 3, 2012, at 2:39 PM, developers-request at admb-project.org wrote:
>
>> Send Developers mailing list submissions to
>> developers at admb-project.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://lists.admb-project.org/mailman/listinfo/developers
>> or, via email, send a message with subject or body 'help' to
>> developers-request at admb-project.org
>>
>> You can reach the person managing the list at
>> developers-owner at admb-project.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Developers digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: admb threading (Matthew Supernaw)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 3 Dec 2012 14:29:01 -0500
>> From: Matthew Supernaw <matthew.supernaw at noaa.gov>
>> To: developers at admb-project.org
>> Subject: Re: [Developers] admb threading
>> Message-ID: <C8E76651-698E-4D98-A995-40A2F259B069 at noaa.gov>
>> Content-Type: text/plain; charset="us-ascii"
>>
>>
>> Perhaps making admb type containers thread safe is a good place to start. This would make it easier to modify admb internally as well as allow end users to make their own applications concurrent.
>>
>> I've started working on this issue for one of our staff. After profiling his model, I found nearly 70-80% of the runtime was in his user_function looping through a df1b2matrix and doing operations on its elements. In this particular case,
>> admb is quite fast and is really just waiting for user_function evaluations(nested for loops). This is the perfect location for multi-threading.
>>
>>
>> At the moment I'm working on a thread safe wrapper for df1b2matrix in order to speed up this particular model. This should be all that is needed for this particular case, but I suspect this cause may be a common.
>>
>>
>> Attached is a simple platform independent threading library.
>>
>>
>> My strategy for making df1b2matrix thread safe is to wrap it as follows:
>>
>>
>> 11 #include <admodel.h>
>> 12 #include "Threads.hpp"
>> 13
>> 14
>> 15 namespace admb {
>> 16
>> 17 class ts_df1b2matrix {
>> 18
>> 19 int allocated(void) {
>> 20 noaa::threads::Lock(mutex_);
>> 21 df1b2matrix_.allocated();
>> 22 }
>> 23
>> 24 void initialize(void) {
>> 25 noaa::threads::Lock(mutex_);
>> 26 df1b2matrix_.initialize();
>> 27
>> 28 }
>> 29
>> 30 ~ts_df1b2matrix() {
>> 31 noaa::threads::Lock(mutex_);
>> 32 ~df1b2matrix_;
>> 33 }
>> 34
>> 35 int rowmin(void) const {
>> 36 noaa::threads::Lock(mutex_);
>> 37 df1b2matrix_.rowmin();
>> 38 }
>> 39
>> 40 int indexmin(void) const {
>> 41 noaa::threads::Lock(mutex_);
>> 42 df1b2matrix_.indexmin();
>> 43 }
>> 44
>> 45 int indexmax(void) const {
>> 46 noaa::threads::Lock(mutex_);
>> 47 df1b2matrix_.indexmax();
>> 48 }
>> 49
>> 50 int rowmax(void) const {
>> 51 noaa::threads::Lock(mutex_);
>> 52 df1b2matrix_.rowmax();
>> 53 }
>> 54
>> 55 int size(void) const {
>> 56 noaa::threads::Lock(mutex_);
>> 57 df1b2matrix_.size();
>> 58 }
>> 59
>> 60 ts_df1b2matrix(int nrl, int nrh) {
>> 61 noaa::threads::Lock(mutex_);
>> 62 df1b2matrix_(nrl, nrh);
>> 63 }
>> 64
>> 65 ts_df1b2matrix(const df1b2matrix &other) {
>> 66 noaa::threads::Lock(mutex_);
>> 67 df1b2matrix_(other);
>> 68 }
>> 69
>> 70 ts_df1b2matrix(const ts_df1b2matrix &other) {
>> 71 noaa::threads::Lock(mutex_);
>> 72 noaa::threads::Lock2(other.mutex_);
>> 73 df1b2matrix_(other.df1b2matrix_);
>> 74 }
>> 75
>> 76 ts_df1b2matrix(int nrl, int nrh, const index_type &ncl, const index_type &nch) {
>> 77 noaa::threads::Lock(mutex_);
>> 78 df1b2matrix(nrl, nrh, ncl, nch);
>> 79 }
>> 80
>> 81 ts_df1b2matrix& operator=(const df3_one_matrix &other) {
>> 82 noaa::threads::Lock(mutex_);
>> 83 this->df1b2matrix_ = other;
>> 84 }
>> 85
>> .
>> .
>> .
>> .
>>
>> 162
>> 163 private:
>> 164 mutable noaa::threads::Mutex mutex_;
>> 165 df1b2matrix df1b2matrix_;
>> 166
>> 167
>> 168 };
>>
>>
>> The call "noaa::threads::Lock(mutex_); " will lock the function and then unlock it as it goes out of scope.
>> I'll be happy to pass the finished product for review.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Matthew Supernaw
>> Scientific Programmer
>> National Oceanic and Atmospheric Administration
>> National Marine Fisheries Service
>> Sustainable Fisheries Division
>> St. Petersburg, FL, 33701
>> Office 727-551-5606
>> Fax 727-824-5300
>>
>> On Nov 29, 2012, at 3:00 PM, developers-request at admb-project.org wrote:
>>
>>> Send Developers mailing list submissions to
>>> developers at admb-project.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> http://lists.admb-project.org/mailman/listinfo/developers
>>> or, via email, send a message with subject or body 'help' to
>>> developers-request at admb-project.org
>>>
>>> You can reach the person managing the list at
>>> developers-owner at admb-project.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Developers digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Re: admb threading (Mark Maunder)
>>> 2. Re: admb threading (dave fournier)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Thu, 29 Nov 2012 17:50:41 +0000
>>> From: Mark Maunder <mmaunder at iattc.org>
>>> To: dave fournier <davef at otter-rsch.com>,
>>> "developers at admb-project.org" <developers at admb-project.org>
>>> Subject: Re: [Developers] admb threading
>>> Message-ID:
>>> <339913E1960AE142A9373DFCD849F3DA325FA148 at mail1.lajolla.iattc.org>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> What about the calculation of the hessian, which can be quite long on parameter rich models.
>>> Profile likelihoods would also be another easy one
>>>
>>>
>>> -----Original Message-----
>>> From: developers-bounces at admb-project.org [mailto:developers-bounces at admb-project.org] On Behalf Of dave fournier
>>> Sent: Thursday, November 29, 2012 9:26 AM
>>> To: developers at admb-project.org
>>> Subject: Re: [Developers] admb threading
>>>
>>> On 12-11-29 09:11 AM, Hans J. Skaug wrote:
>>>
>>> The obvious transparent one is the -ndb (num der blocks) which was already set up for mult-threading, and I recall Derek was doing something with that, but I never heard about it again, and it is not for separable models. For separable models one could split up the separable function calls by different threads in a transparent manner. Both of these involve using the __thread declaration to deal with some global data structures. The real point of my proof of concept example was to demonstrate that this can be done quite easily.
>>>
>>>
>>>
>>>> Both are useful, but currently "transparent to the user" is the most important.
>>>>
>>>> hans
>>>>
>>>>> -----Original Message-----
>>>>> From: developers-bounces at admb-project.org [mailto:developers-
>>>>> bounces at admb-project.org] On Behalf Of Mark Maunder
>>>>> Sent: Thursday, November 29, 2012 8:23 AM
>>>>> To: John Sibert; ADMB Developers
>>>>> Subject: Re: [Developers] admb threading
>>>>>
>>>>> parallel code that is "transparent" to the user
>>>>>
>>>>> -----Original Message-----
>>>>> From: developers-bounces at admb-project.org [mailto:developers-
>>>>> bounces at admb-project.org] On Behalf Of John Sibert
>>>>> Sent: Wednesday, November 28, 2012 4:30 PM
>>>>> To: ADMB Developers
>>>>> Subject: [Developers] admb threading
>>>>>
>>>>> Johnoel and I need some feedback about how to approach threading.
>>>>> Dave has provided a nice proof of concept using pthreads to implement
>>>>> parallel processing on larger chunks of code. This approach is likely
>>>>> to have the biggest performance improvement, but seems application
>>>>> specific and would require more expertize on the part of users.
>>>>>
>>>>> Alternatively it is possible to implement threading internally in the
>>>>> ADMB libraries, concentrating on smaller chunks of code, for instance
>>>>> the solve(...) function. This approach would probably have smaller
>>>>> performance payoff in most applications, but would be more transparent to users.
>>>>>
>>>>> In principle, the two approaches are not mutually exclusive.
>>>>>
>>>>> So my question to the ADMB Developer group is what did we mean when
>>>>> we assigned a high priority to parallelization? Do we want parallel
>>>>> code that is "transparent" to the user (if so what parts of the would
>>>>> have the highest priority)? Or do we want to develop tools that allow
>>>>> users to create their on threaded code for specific applications?
>>>>> (Don't tell me both.)
>>>>>
>>>>> Cheers,
>>>>> John
>>>>> PS enjoy the attached.
>>>>>
>>>>> --
>>>>> John Sibert
>>>>> Emeritus Researcher, SOEST
>>>>> University of Hawaii at Manoa
>>>>>
>>>>> Visit the ADMB project http://admb-project.org/
>>>>>
>>>>> _______________________________________________
>>>>> Developers mailing list
>>>>> Developers at admb-project.org
>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>> _______________________________________________
>>>> Developers mailing list
>>>> Developers at admb-project.org
>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>
>>> _______________________________________________
>>> Developers mailing list
>>> Developers at admb-project.org
>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Thu, 29 Nov 2012 09:53:45 -0800
>>> From: dave fournier <davef at otter-rsch.com>
>>> To: Mark Maunder <mmaunder at iattc.org>
>>> Cc: "developers at admb-project.org" <developers at admb-project.org>
>>> Subject: Re: [Developers] admb threading
>>> Message-ID: <50B7A129.5030200 at otter-rsch.com>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> On 12-11-29 09:50 AM, Mark Maunder wrote:
>>>
>>> The biggest improvement to the profile likelihood would be to replace
>>> the current
>>> penalty function method with the augmented Lagrangian.
>>>
>>>> What about the calculation of the hessian, which can be quite long on parameter rich models.
>>>> Profile likelihoods would also be another easy one
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: developers-bounces at admb-project.org [mailto:developers-bounces at admb-project.org] On Behalf Of dave fournier
>>>> Sent: Thursday, November 29, 2012 9:26 AM
>>>> To: developers at admb-project.org
>>>> Subject: Re: [Developers] admb threading
>>>>
>>>> On 12-11-29 09:11 AM, Hans J. Skaug wrote:
>>>>
>>>> The obvious transparent one is the -ndb (num der blocks) which was already set up for mult-threading, and I recall Derek was doing something with that, but I never heard about it again, and it is not for separable models. For separable models one could split up the separable function calls by different threads in a transparent manner. Both of these involve using the __thread declaration to deal with some global data structures. The real point of my proof of concept example was to demonstrate that this can be done quite easily.
>>>>
>>>>
>>>>
>>>>> Both are useful, but currently "transparent to the user" is the most important.
>>>>>
>>>>> hans
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: developers-bounces at admb-project.org [mailto:developers-
>>>>>> bounces at admb-project.org] On Behalf Of Mark Maunder
>>>>>> Sent: Thursday, November 29, 2012 8:23 AM
>>>>>> To: John Sibert; ADMB Developers
>>>>>> Subject: Re: [Developers] admb threading
>>>>>>
>>>>>> parallel code that is "transparent" to the user
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: developers-bounces at admb-project.org [mailto:developers-
>>>>>> bounces at admb-project.org] On Behalf Of John Sibert
>>>>>> Sent: Wednesday, November 28, 2012 4:30 PM
>>>>>> To: ADMB Developers
>>>>>> Subject: [Developers] admb threading
>>>>>>
>>>>>> Johnoel and I need some feedback about how to approach threading.
>>>>>> Dave has provided a nice proof of concept using pthreads to implement
>>>>>> parallel processing on larger chunks of code. This approach is likely
>>>>>> to have the biggest performance improvement, but seems application
>>>>>> specific and would require more expertize on the part of users.
>>>>>>
>>>>>> Alternatively it is possible to implement threading internally in the
>>>>>> ADMB libraries, concentrating on smaller chunks of code, for instance
>>>>>> the solve(...) function. This approach would probably have smaller
>>>>>> performance payoff in most applications, but would be more transparent to users.
>>>>>>
>>>>>> In principle, the two approaches are not mutually exclusive.
>>>>>>
>>>>>> So my question to the ADMB Developer group is what did we mean when
>>>>>> we assigned a high priority to parallelization? Do we want parallel
>>>>>> code that is "transparent" to the user (if so what parts of the would
>>>>>> have the highest priority)? Or do we want to develop tools that allow
>>>>>> users to create their on threaded code for specific applications?
>>>>>> (Don't tell me both.)
>>>>>>
>>>>>> Cheers,
>>>>>> John
>>>>>> PS enjoy the attached.
>>>>>>
>>>>>> --
>>>>>> John Sibert
>>>>>> Emeritus Researcher, SOEST
>>>>>> University of Hawaii at Manoa
>>>>>>
>>>>>> Visit the ADMB project http://admb-project.org/
>>>>>>
>>>>>> _______________________________________________
>>>>>> Developers mailing list
>>>>>> Developers at admb-project.org
>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>> _______________________________________________
>>>>> Developers mailing list
>>>>> Developers at admb-project.org
>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>> _______________________________________________
>>>> Developers mailing list
>>>> Developers at admb-project.org
>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Developers mailing list
>>> Developers at admb-project.org
>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>
>>>
>>> End of Developers Digest, Vol 45, Issue 5
>>> *****************************************
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.admb-project.org/pipermail/developers/attachments/20121203/086c40d2/attachment.html>
>> -------------- next part --------------
>> A non-text attachment was scrubbed...
>> Name: Threads.hpp
>> Type: application/octet-stream
>> Size: 27706 bytes
>> Desc: not available
>> URL: <http://lists.admb-project.org/pipermail/developers/attachments/20121203/086c40d2/attachment.obj>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://lists.admb-project.org/pipermail/developers/attachments/20121203/086c40d2/attachment-0001.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> Developers mailing list
>> Developers at admb-project.org
>> http://lists.admb-project.org/mailman/listinfo/developers
>>
>>
>> End of Developers Digest, Vol 46, Issue 3
>> *****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.admb-project.org/pipermail/developers/attachments/20121204/41a0a0bb/attachment-0001.html>
More information about the Developers
mailing list