From SteveM at iphc.int Wed Oct 1 12:20:58 2014 From: SteveM at iphc.int (Steve Martell) Date: Wed, 1 Oct 2014 19:20:58 +0000 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: References: Message-ID: <2AC6FFF0-E347-4219-A5A2-D8C7326ED4CF@iphc.int> Hola Carlos, Mucho gracias por la correccion. I will update the source code. Steve On Oct 1, 2014, at 10:59 AM, Carlos Martorell > wrote: Hi, After checking the output of an ADMB run I believe that I have found a problem in (at least) a couple of the functions used to calculate the negative log-likelihoods in dpois and dnbinom functions in the statsLib.h library. When a vector of parameters (lambdas in the Poisson case) is provided, the code for the log-likelihood is 00067 dvariable dpois(const dvector& k, const dvar_vector& lambda) 00068 { 00069 RETURN_ARRAYS_INCREMENT(); 00070 int i; 00071 int n = size_count(k); 00072 dvariable nll=0; 00073 for(i = 1; i <= n; i++) 00074 { 00075 nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); 00076 } 00077 RETURN_ARRAYS_DECREMENT(); 00078 return nll; 00079 } I believe that the correct code for line 00075 should be nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); This is evident in the same function when lambda is constant, e.g., 00035 dvariable dpois(const double& k, const prevariable& lambda) 00036 { 00037 RETURN_ARRAYS_INCREMENT(); 00038 dvariable tmp = -k*log(lambda)+lambda + gammln(k+1.); 00039 RETURN_ARRAYS_DECREMENT(); 00040 return tmp; 00041 } I found these codes both in the ADMB version that I downloaded and in the package documentation (http://admb-project.org/documentation/api/dpois_8cpp_source.html). After making the correction that I propose above and re-running ADMB, the program reports the correct log-likelihoods. The same problem can be found in dnbinom and maybe in other functions. Given that the functions provided for the estimation of negative log-likelihoods are widely used, it would be important to check that their outputs are correct. Best, -- I think [the island's] unusually sterile character gives it a grandeur which more vegetation might have spoiled. ----Ch. Darwin. El diario del Beagle Dr. Carlos Martorell Departamento de Ecolog?a y Recursos Naturales Facultad de Ciencias-UNAM Circuito Exterior S/N, Cd. Universitaria 04510 M?xico D.F. Tel. (55) 5622 4835 Fax (55) 5622 4828 http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaguar.smart at gmail.com Wed Oct 1 12:05:37 2014 From: jaguar.smart at gmail.com (bigA) Date: Wed, 1 Oct 2014 12:05:37 -0700 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: References: Message-ID: If I am not mistaken, in Iceland, there was substantial discussion amongst developers about what functions returned log-likelihood and which functions returned negative log-likelihood. If I am not mistaken this hasn't been standardized; it is often different from R. The difference between the two versions of 00075 appears to simply be a negative sign? Is this a documentation failure or a bug? Aaron nll -= -k(i)*log (lambda(i))+lambda(i)+gammln (k(i)+1.); vs nll -= k(i)*log (lambda(i))+lambda(i)+gammln (k(i)+1.); On Tue, Sep 30, 2014 at 6:57 PM, Carlos Martorell < martorell at ciencias.unam.mx> wrote: > Hi, > After checking the output of an ADMB run I believe that I have found a > problem in (at least) a couple of the functions used to calculate the > negative log-likelihoods in dpois and dnbinom functions in the statsLib.h > library. When a vector of parameters (lambdas in the Poisson case) is > provided, the code for the log-likelihood is > > 00067 dvariable dpois (const dvector & k, const dvar_vector & lambda)00068 {00069 RETURN_ARRAYS_INCREMENT ();00070 int i;00071 int n = size_count (k);00072 dvariable nll=0;00073 for(i = 1; i <= n; i++)00074 {00075 nll -= k(i)*log (lambda(i))+lambda(i)+gammln (k(i)+1.);00076 }00077 RETURN_ARRAYS_DECREMENT ();00078 return nll;00079 } > > I believe that the correct code for line 00075 should be > > nll -= -k(i)*log > > (lambda(i))+lambda(i)+gammln > > (k(i)+1.); > > This is evident in the same function when lambda is constant, e.g., > > 00035 dvariable dpois (const double& k, const prevariable & lambda)00036 {00037 RETURN_ARRAYS_INCREMENT ();00038 dvariable tmp = -k*log (lambda)+lambda + gammln (k+1.);00039 RETURN_ARRAYS_DECREMENT ();00040 return tmp;00041 } > > I found these codes both in the ADMB version that I downloaded and in the > package documentation ( > http://admb-project.org/documentation/api/dpois_8cpp_source.html). After > making the correction that I propose above and re-running ADMB, the program > reports the correct log-likelihoods. The same problem can be found in > dnbinom and maybe in other functions. > Given that the functions provided for the estimation of negative > log-likelihoods are widely used, it would be important to check that their > outputs are correct. > Best, > > > > -- > I think [the island's] unusually sterile character gives it a > grandeur > which more vegetation might have spoiled. > ----Ch. Darwin. El > diario del Beagle > > Dr. Carlos Martorell > > Departamento de Ecolog?a y Recursos Naturales > Facultad de Ciencias-UNAM > Circuito Exterior S/N, Cd. Universitaria > 04510 M?xico D.F. > Tel. (55) 5622 4835 Fax (55) 5622 4828 > > http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl > > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Wed Oct 1 12:28:20 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Wed, 1 Oct 2014 09:28:20 -1000 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: <2AC6FFF0-E347-4219-A5A2-D8C7326ED4CF@iphc.int> References: <2AC6FFF0-E347-4219-A5A2-D8C7326ED4CF@iphc.int> Message-ID: Thanks Steve... On Wed, Oct 1, 2014 at 9:20 AM, Steve Martell wrote: > Hola Carlos, > > Mucho gracias por la correccion. I will update the source code. > > Steve > On Oct 1, 2014, at 10:59 AM, Carlos Martorell > wrote: > > Hi, > After checking the output of an ADMB run I believe that I have found a > problem in (at least) a couple of the functions used to calculate the > negative log-likelihoods in dpois and dnbinom functions in the statsLib.h > library. When a vector of parameters (lambdas in the Poisson case) is > provided, the code for the log-likelihood is > > 00067 dvariable dpois (const dvector & k, const dvar_vector & lambda)00068 {00069 RETURN_ARRAYS_INCREMENT ();00070 int i;00071 int n = size_count (k);00072 dvariable nll=0;00073 for(i = 1; i <= n; i++)00074 {00075 nll -= k(i)*log (lambda(i))+lambda(i)+gammln (k(i)+1.);00076 }00077 RETURN_ARRAYS_DECREMENT ();00078 return nll;00079 } > > I believe that the correct code for line 00075 should be > > nll -= -k(i)*log > > (lambda(i))+lambda(i)+gammln > > (k(i)+1.); > > This is evident in the same function when lambda is constant, e.g., > > 00035 dvariable dpois (const double& k, const prevariable & lambda)00036 {00037 RETURN_ARRAYS_INCREMENT ();00038 dvariable tmp = -k*log (lambda)+lambda + gammln (k+1.);00039 RETURN_ARRAYS_DECREMENT ();00040 return tmp;00041 } > > I found these codes both in the ADMB version that I downloaded and in > the package documentation ( > http://admb-project.org/documentation/api/dpois_8cpp_source.html). After > making the correction that I propose above and re-running ADMB, the program > reports the correct log-likelihoods. The same problem can be found in > dnbinom and maybe in other functions. > Given that the functions provided for the estimation of negative > log-likelihoods are widely used, it would be important to check that their > outputs are correct. > Best, > > > > -- > I think [the island's] unusually sterile character gives it a > grandeur > which more vegetation might have spoiled. > ----Ch. Darwin. El > diario del Beagle > > Dr. Carlos Martorell > > Departamento de Ecolog?a y Recursos Naturales > Facultad de Ciencias-UNAM > Circuito Exterior S/N, Cd. Universitaria > 04510 M?xico D.F. > Tel. (55) 5622 4835 Fax (55) 5622 4828 > > http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > > > > ------------------------------ > > This internet e-mail message, and any files transmitted with it, contains > confidential, privileged information that is intended only for the > addressee. If you have received this e-mail message in error, please call > us at (206) 634-1838 collect if necessary) and ask to speak to the > message sender. Nothing in this e-mail or the act of transmitting it, is to > be construed as a waiver of any rights or privileges enjoyed by the sender > or the International Pacific Halibut Commission pursuant to the > International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. > > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mintoc at gmail.com Thu Oct 2 03:04:28 2014 From: mintoc at gmail.com (Coilin Minto) Date: Thu, 2 Oct 2014 10:04:28 +0000 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: <2AC6FFF0-E347-4219-A5A2-D8C7326ED4CF@iphc.int> References: <2AC6FFF0-E347-4219-A5A2-D8C7326ED4CF@iphc.int> Message-ID: Hi Carlos, Just a clarification on the fix. If it's the negative log-likelihood, it should be nll += -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); The suggested nll -= -k(i)*log (lambda(i))+lambda(i)+gammln (k(i)+1.); returns the log-likelihood. Cheers, Coilin On Wed, Oct 1, 2014 at 7:20 PM, Steve Martell wrote: > Hola Carlos, > > Mucho gracias por la correccion. I will update the source code. > > Steve > On Oct 1, 2014, at 10:59 AM, Carlos Martorell > wrote: > > Hi, > After checking the output of an ADMB run I believe that I have found a > problem in (at least) a couple of the functions used to calculate the > negative log-likelihoods in dpois and dnbinom functions in the statsLib.h > library. When a vector of parameters (lambdas in the Poisson case) is > provided, the code for the log-likelihood is > > 00067 dvariable dpois (const dvector & k, const dvar_vector & lambda)00068 {00069 RETURN_ARRAYS_INCREMENT ();00070 int i;00071 int n = size_count (k);00072 dvariable nll=0;00073 for(i = 1; i <= n; i++)00074 {00075 nll -= k(i)*log (lambda(i))+lambda(i)+gammln (k(i)+1.);00076 }00077 RETURN_ARRAYS_DECREMENT ();00078 return nll;00079 } > > I believe that the correct code for line 00075 should be > > nll -= -k(i)*log > > (lambda(i))+lambda(i)+gammln > > (k(i)+1.); > > This is evident in the same function when lambda is constant, e.g., > > 00035 dvariable dpois (const double& k, const prevariable & lambda)00036 {00037 RETURN_ARRAYS_INCREMENT ();00038 dvariable tmp = -k*log (lambda)+lambda + gammln (k+1.);00039 RETURN_ARRAYS_DECREMENT ();00040 return tmp;00041 } > > I found these codes both in the ADMB version that I downloaded and in > the package documentation ( > http://admb-project.org/documentation/api/dpois_8cpp_source.html). After > making the correction that I propose above and re-running ADMB, the program > reports the correct log-likelihoods. The same problem can be found in > dnbinom and maybe in other functions. > Given that the functions provided for the estimation of negative > log-likelihoods are widely used, it would be important to check that their > outputs are correct. > Best, > > > > -- > I think [the island's] unusually sterile character gives it a > grandeur > which more vegetation might have spoiled. > ----Ch. Darwin. El > diario del Beagle > > Dr. Carlos Martorell > > Departamento de Ecolog?a y Recursos Naturales > Facultad de Ciencias-UNAM > Circuito Exterior S/N, Cd. Universitaria > 04510 M?xico D.F. > Tel. (55) 5622 4835 Fax (55) 5622 4828 > > http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > > > > ------------------------------ > > This internet e-mail message, and any files transmitted with it, contains > confidential, privileged information that is intended only for the > addressee. If you have received this e-mail message in error, please call > us at (206) 634-1838 collect if necessary) and ask to speak to the > message sender. Nothing in this e-mail or the act of transmitting it, is to > be construed as a waiver of any rights or privileges enjoyed by the sender > or the International Pacific Halibut Commission pursuant to the > International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. > > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From SteveM at iphc.int Thu Oct 2 11:06:05 2014 From: SteveM at iphc.int (Steve Martell) Date: Thu, 2 Oct 2014 18:06:05 +0000 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: References: Message-ID: <95C15817-3CD8-4714-8C6E-5C2D6875D806@iphc.int> All the functions in statsLib return the negative log likelihood. ADMB minimizes the negative loglikelihoods. It does not maximize a function. So for me it seemed obvious that the functions should return the negative loglikelihood. But that was a choice I made. If someone wants to change it, the open source platform allows you to do just that. But if you do change it, please warn all others who use these functions to reverse the sign in their objective functions. There was a bug in the vectorized version of dpois, so I fixed it. I very much appreciate Carlos pointing the bug out. I guess this is how open source software is supposed to work. Grumpy Steve On Oct 2, 2014, at 10:56 AM, bigA > wrote: If I am not mistaken, in Iceland, there was substantial discussion amongst developers about what functions returned log-likelihood and which functions returned negative log-likelihood. If I am not mistaken this hasn't been standardized; it is often different from R. The difference between the two versions of 00075 appears to simply be a negative sign? Is this a documentation failure or a bug? Aaron nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); vs nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); On Tue, Sep 30, 2014 at 6:57 PM, Carlos Martorell > wrote: Hi, After checking the output of an ADMB run I believe that I have found a problem in (at least) a couple of the functions used to calculate the negative log-likelihoods in dpois and dnbinom functions in the statsLib.h library. When a vector of parameters (lambdas in the Poisson case) is provided, the code for the log-likelihood is 00067 dvariable dpois(const dvector& k, const dvar_vector& lambda) 00068 { 00069 RETURN_ARRAYS_INCREMENT(); 00070 int i; 00071 int n = size_count(k); 00072 dvariable nll=0; 00073 for(i = 1; i <= n; i++) 00074 { 00075 nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); 00076 } 00077 RETURN_ARRAYS_DECREMENT(); 00078 return nll; 00079 } I believe that the correct code for line 00075 should be nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); This is evident in the same function when lambda is constant, e.g., 00035 dvariable dpois(const double& k, const prevariable& lambda) 00036 { 00037 RETURN_ARRAYS_INCREMENT(); 00038 dvariable tmp = -k*log(lambda)+lambda + gammln(k+1.); 00039 RETURN_ARRAYS_DECREMENT(); 00040 return tmp; 00041 } I found these codes both in the ADMB version that I downloaded and in the package documentation (http://admb-project.org/documentation/api/dpois_8cpp_source.html). After making the correction that I propose above and re-running ADMB, the program reports the correct log-likelihoods. The same problem can be found in dnbinom and maybe in other functions. Given that the functions provided for the estimation of negative log-likelihoods are widely used, it would be important to check that their outputs are correct. Best, -- I think [the island's] unusually sterile character gives it a grandeur which more vegetation might have spoiled. ----Ch. Darwin. El diario del Beagle Dr. Carlos Martorell Departamento de Ecolog?a y Recursos Naturales Facultad de Ciencias-UNAM Circuito Exterior S/N, Cd. Universitaria 04510 M?xico D.F. Tel. (55) 5622 4835 Fax (55) 5622 4828 http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmaunder at iattc.org Thu Oct 2 11:10:36 2014 From: mmaunder at iattc.org (Mark Maunder) Date: Thu, 2 Oct 2014 18:10:36 +0000 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: <95C15817-3CD8-4714-8C6E-5C2D6875D806@iphc.int> References: <95C15817-3CD8-4714-8C6E-5C2D6875D806@iphc.int> Message-ID: <339913E1960AE142A9373DFCD849F3DA53A61901@mail1.lajolla.iattc.org> Should there be a naming convention so that nll_xxxx gives a negative log likelihood lnl_xxxx gives a log likelihood R names gives a likelihood or distribution And it is up to the user to choose the nll for an objective function? From: users-bounces at admb-project.org [mailto:users-bounces at admb-project.org] On Behalf Of Steve Martell Sent: Thursday, October 02, 2014 11:06 AM To: Aaron Greenberg Cc: users at admb-project.org; Carlos Martorell Subject: Re: [ADMB Users] Problem with negative log likelihoods? All the functions in statsLib return the negative log likelihood. ADMB minimizes the negative loglikelihoods. It does not maximize a function. So for me it seemed obvious that the functions should return the negative loglikelihood. But that was a choice I made. If someone wants to change it, the open source platform allows you to do just that. But if you do change it, please warn all others who use these functions to reverse the sign in their objective functions. There was a bug in the vectorized version of dpois, so I fixed it. I very much appreciate Carlos pointing the bug out. I guess this is how open source software is supposed to work. Grumpy Steve On Oct 2, 2014, at 10:56 AM, bigA > wrote: If I am not mistaken, in Iceland, there was substantial discussion amongst developers about what functions returned log-likelihood and which functions returned negative log-likelihood. If I am not mistaken this hasn't been standardized; it is often different from R. The difference between the two versions of 00075 appears to simply be a negative sign? Is this a documentation failure or a bug? Aaron nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); vs nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); On Tue, Sep 30, 2014 at 6:57 PM, Carlos Martorell > wrote: Hi, After checking the output of an ADMB run I believe that I have found a problem in (at least) a couple of the functions used to calculate the negative log-likelihoods in dpois and dnbinom functions in the statsLib.h library. When a vector of parameters (lambdas in the Poisson case) is provided, the code for the log-likelihood is 00067 dvariable dpois(const dvector& k, const dvar_vector& lambda) 00068 { 00069 RETURN_ARRAYS_INCREMENT(); 00070 int i; 00071 int n = size_count(k); 00072 dvariable nll=0; 00073 for(i = 1; i <= n; i++) 00074 { 00075 nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); 00076 } 00077 RETURN_ARRAYS_DECREMENT(); 00078 return nll; 00079 } I believe that the correct code for line 00075 should be nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); This is evident in the same function when lambda is constant, e.g., 00035 dvariable dpois(const double& k, const prevariable& lambda) 00036 { 00037 RETURN_ARRAYS_INCREMENT(); 00038 dvariable tmp = -k*log(lambda)+lambda + gammln(k+1.); 00039 RETURN_ARRAYS_DECREMENT(); 00040 return tmp; 00041 } I found these codes both in the ADMB version that I downloaded and in the package documentation (http://admb-project.org/documentation/api/dpois_8cpp_source.html). After making the correction that I propose above and re-running ADMB, the program reports the correct log-likelihoods. The same problem can be found in dnbinom and maybe in other functions. Given that the functions provided for the estimation of negative log-likelihoods are widely used, it would be important to check that their outputs are correct. Best, -- I think [the island's] unusually sterile character gives it a grandeur which more vegetation might have spoiled. ----Ch. Darwin. El diario del Beagle Dr. Carlos Martorell Departamento de Ecolog?a y Recursos Naturales Facultad de Ciencias-UNAM Circuito Exterior S/N, Cd. Universitaria 04510 M?xico D.F. Tel. (55) 5622 4835 Fax (55) 5622 4828 http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From SteveM at iphc.int Thu Oct 2 11:19:00 2014 From: SteveM at iphc.int (Steve Martell) Date: Thu, 2 Oct 2014 18:19:00 +0000 Subject: [ADMB Users] Problem with negative log likelihoods? In-Reply-To: <339913E1960AE142A9373DFCD849F3DA53A61901@mail1.lajolla.iattc.org> References: <95C15817-3CD8-4714-8C6E-5C2D6875D806@iphc.int> <339913E1960AE142A9373DFCD849F3DA53A61901@mail1.lajolla.iattc.org> Message-ID: Mark, I like you suggestions 1&2. It would be easy to write a wrapper for the existing functions. Not sure how to deal with the existing names. I just dont want to go in and change all the R names, as I fear it would break a lot of code out there. Should have thought this out better 5 years ago when I did this stuff. But perhaps I'm the only one using dnorm etc. S On Oct 2, 2014, at 11:10 AM, Mark Maunder > wrote: Should there be a naming convention so that nll_xxxx gives a negative log likelihood lnl_xxxx gives a log likelihood R names gives a likelihood or distribution And it is up to the user to choose the nll for an objective function? From: users-bounces at admb-project.org [mailto:users-bounces at admb-project.org] On Behalf Of Steve Martell Sent: Thursday, October 02, 2014 11:06 AM To: Aaron Greenberg Cc: users at admb-project.org; Carlos Martorell Subject: Re: [ADMB Users] Problem with negative log likelihoods? All the functions in statsLib return the negative log likelihood. ADMB minimizes the negative loglikelihoods. It does not maximize a function. So for me it seemed obvious that the functions should return the negative loglikelihood. But that was a choice I made. If someone wants to change it, the open source platform allows you to do just that. But if you do change it, please warn all others who use these functions to reverse the sign in their objective functions. There was a bug in the vectorized version of dpois, so I fixed it. I very much appreciate Carlos pointing the bug out. I guess this is how open source software is supposed to work. Grumpy Steve On Oct 2, 2014, at 10:56 AM, bigA > wrote: If I am not mistaken, in Iceland, there was substantial discussion amongst developers about what functions returned log-likelihood and which functions returned negative log-likelihood. If I am not mistaken this hasn't been standardized; it is often different from R. The difference between the two versions of 00075 appears to simply be a negative sign? Is this a documentation failure or a bug? Aaron nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); vs nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); On Tue, Sep 30, 2014 at 6:57 PM, Carlos Martorell > wrote: Hi, After checking the output of an ADMB run I believe that I have found a problem in (at least) a couple of the functions used to calculate the negative log-likelihoods in dpois and dnbinom functions in the statsLib.h library. When a vector of parameters (lambdas in the Poisson case) is provided, the code for the log-likelihood is 00067 dvariable dpois(const dvector& k, const dvar_vector& lambda) 00068 { 00069 RETURN_ARRAYS_INCREMENT(); 00070 int i; 00071 int n = size_count(k); 00072 dvariable nll=0; 00073 for(i = 1; i <= n; i++) 00074 { 00075 nll -= k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); 00076 } 00077 RETURN_ARRAYS_DECREMENT(); 00078 return nll; 00079 } I believe that the correct code for line 00075 should be nll -= -k(i)*log(lambda(i))+lambda(i)+gammln(k(i)+1.); This is evident in the same function when lambda is constant, e.g., 00035 dvariable dpois(const double& k, const prevariable& lambda) 00036 { 00037 RETURN_ARRAYS_INCREMENT(); 00038 dvariable tmp = -k*log(lambda)+lambda + gammln(k+1.); 00039 RETURN_ARRAYS_DECREMENT(); 00040 return tmp; 00041 } I found these codes both in the ADMB version that I downloaded and in the package documentation (http://admb-project.org/documentation/api/dpois_8cpp_source.html). After making the correction that I propose above and re-running ADMB, the program reports the correct log-likelihoods. The same problem can be found in dnbinom and maybe in other functions. Given that the functions provided for the estimation of negative log-likelihoods are widely used, it would be important to check that their outputs are correct. Best, -- I think [the island's] unusually sterile character gives it a grandeur which more vegetation might have spoiled. ----Ch. Darwin. El diario del Beagle Dr. Carlos Martorell Departamento de Ecolog?a y Recursos Naturales Facultad de Ciencias-UNAM Circuito Exterior S/N, Cd. Universitaria 04510 M?xico D.F. Tel. (55) 5622 4835 Fax (55) 5622 4828 http://www.fciencias.unam.mx/investigacion/grupos/biologia/ecologia/ambientesl _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefflaake at gmail.com Thu Oct 2 12:15:11 2014 From: jefflaake at gmail.com (Jeff Laake) Date: Thu, 2 Oct 2014 12:15:11 -0700 Subject: [ADMB Users] Non-matching analytic and numeric derivatives Message-ID: I'm running into a situation in which the numeric and analytic derivatives are vastly different. I have separate code FORTRAN code that fits the model with numeric derivatives and I know that the log-likelihood value produced by the ADMB code and the numeric derivatives are correct because they match the FORTRAN code. The analytic derivatives are several orders of magnitude greater (960 vs 0.0001) so the optimization algorithm thinks it should move but can't from what I can tell. It gives up because it can't make any improvements because I started the code at the MLEs derived from the FORTRAN code. This is a very large problem. It is a Cormack-Jolly-Seber model with 10000+ capture histories for 27 occasions with 251 parameters. The input file is 135MB because the design matrix has 260,000+ rows and 251 columns. If I use a subset of the data (50-75%) the input file is reduced to less than 100MB and everything works fine. Also if I keep the data set the same and reduce the number of columns in the design matrix to 70, it also works fine. It is not a single model that fails. I tried a mdeol with 3/4 of the data and 357 parameters and it fails as well. It appears to be a function of the magnitude of the design matrix (x) which would be in the derivative calculation because the likelihood is a function of 1/(1+exp(-X'Beta)). I do not get any out of error messages. All of the index ranges are input parameters and adjust automatically and I'm compiling in safe mode. Any ideas? Any help is appreciated. I'd like to use ADMB for these problems but may have to remain with my R/FORTRAN code which is slower but works. I'm using ADMB 11 on a Windows machine with 8GB RAM with the Mingw/gcc4.5.2 compiler. regards --jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdavefournier at gmail.com Wed Oct 8 16:50:49 2014 From: xdavefournier at gmail.com (Dave Fournier) Date: Wed, 8 Oct 2014 16:50:49 -0700 Subject: [ADMB Users] Non-matching analytic and numeric derivatives Message-ID: If you send me the files I'll check it out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From davef at otter-rsch.com Fri Oct 10 10:45:29 2014 From: davef at otter-rsch.com (dave fournier) Date: Fri, 10 Oct 2014 10:45:29 -0700 Subject: [ADMB Users] Non-matching analytic and numeric derivatives In-Reply-To: References: Message-ID: <54381B39.4010306@otter-rsch.com> On 10/09/2014 07:32 AM, Jeff Laake wrote: This might be of interest to others so I cc's it to the list. In your code for the log-likelihood you have the line. f-= log(pch+0.000000000000000000000001); Now I figured that 0.000000000000000000000001 might be too small. What would happen is that is pch is zero you will get derivatives on the order of 1.0/ (0.000000000000000000000001) and this could make the model numerically unstable at least during the initial fitting stage. I modified this to f-= log(pch+1.e-8); and ran the model for a while and then used the values as initial values to run the model with your original code. It converged easily. There is nothing magical about 1.e-8. For some models you might need something like 1.e-4 to start. Dave > Dave- > > Per Johnoel's request I forwarded a zip containing an example in which > the numeric and analytic derivatives don't match when I use -dd. The > code fails because it thinks it should move but can't make any > progress because it is at the min. > > --jeff > > From bbolker at gmail.com Fri Oct 10 10:53:12 2014 From: bbolker at gmail.com (Ben Bolker) Date: Fri, 10 Oct 2014 13:53:12 -0400 Subject: [ADMB Users] Non-matching analytic and numeric derivatives In-Reply-To: <54381B39.4010306@otter-rsch.com> References: <54381B39.4010306@otter-rsch.com> Message-ID: 1e-8 is a *little* bit special/magical, because it's close to the square root of the minimum representable deviation for a double-precision floating point value -- it's used a lot as a default precision in a certain well-known language & environment for statistical computing ... On Fri, Oct 10, 2014 at 1:45 PM, dave fournier wrote: > On 10/09/2014 07:32 AM, Jeff Laake wrote: > > This might be of interest to others so I cc's it to the list. > > In your code for the log-likelihood you have the line. > > f-= log(pch+0.000000000000000000000001); > > Now I figured that 0.000000000000000000000001 might be too small. > > What would happen is that is pch is zero you will get derivatives on the > order of > > 1.0/ (0.000000000000000000000001) > > and this could make the model numerically unstable at least during the > initial fitting stage. > > I modified this to > > f-= log(pch+1.e-8); > > and ran the model for a while and then used the values as initial values > to run the > model with your original code. It converged easily. > > There is nothing magical about 1.e-8. For some models you might need > something like 1.e-4 to start. > > Dave > > Dave- >> >> Per Johnoel's request I forwarded a zip containing an example in which >> the numeric and analytic derivatives don't match when I use -dd. The >> code fails because it thinks it should move but can't make any >> progress because it is at the min. >> >> --jeff >> >> >> > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Fri Oct 10 10:51:08 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Fri, 10 Oct 2014 07:51:08 -1000 Subject: [ADMB Users] Non-matching analytic and numeric derivatives In-Reply-To: <54381B39.4010306@otter-rsch.com> References: <54381B39.4010306@otter-rsch.com> Message-ID: Thanks Dave! On Fri, Oct 10, 2014 at 7:45 AM, dave fournier wrote: > On 10/09/2014 07:32 AM, Jeff Laake wrote: > > This might be of interest to others so I cc's it to the list. > > In your code for the log-likelihood you have the line. > > f-= log(pch+0.000000000000000000000001); > > Now I figured that 0.000000000000000000000001 might be too small. > > What would happen is that is pch is zero you will get derivatives on the > order of > > 1.0/ (0.000000000000000000000001) > > and this could make the model numerically unstable at least during the > initial fitting stage. > > I modified this to > > f-= log(pch+1.e-8); > > and ran the model for a while and then used the values as initial values > to run the > model with your original code. It converged easily. > > There is nothing magical about 1.e-8. For some models you might need > something like 1.e-4 to start. > > Dave > > > Dave- >> >> Per Johnoel's request I forwarded a zip containing an example in which >> the numeric and analytic derivatives don't match when I use -dd. The >> code fails because it thinks it should move but can't make any >> progress because it is at the min. >> >> --jeff >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmaunder at iattc.org Tue Oct 14 13:58:15 2014 From: mmaunder at iattc.org (Mark Maunder) Date: Tue, 14 Oct 2014 20:58:15 +0000 Subject: [ADMB Users] user-contributed libraries In-Reply-To: References: Message-ID: From: Carolina Minte-Vera Sent: Tuesday, October 14, 2014 11:40 AM To: 'users at admb-project.org' Cc: Arni Magnusson (arnima at hafro.is) Subject: user-contributed libraries Dear Group and Dear Arni, I just installed the binary distribution admb-ide-101-win64.zip from ADMB at Google Code. I want to use some user-contributed libraries in my code such as qfc_est.cpp from http://admb-project.org/documentation/api/group__CONTRIB.html I could not find them in the admb-ide 10.1 I just installed. I would appreciate any hint on how to make them available for the ADMB version I am using. Just manually copy them? Thank you! Carolina Minte-Vera IATTC -------------- next part -------------- An HTML attachment was scrubbed... URL: From SteveM at iphc.int Tue Oct 14 14:06:08 2014 From: SteveM at iphc.int (Steve Martell) Date: Tue, 14 Oct 2014 21:06:08 +0000 Subject: [ADMB Users] user-contributed libraries In-Reply-To: References: Message-ID: <1A113C62-90F3-4EAC-B45D-1AC54B6B5B2B@iphc.int> Quick and dirty, put a copy of qfc_est.cpp in your tpl directory. Then in the GLOBALS_SECTION use #include "qfc_est.cpp". When you compile the code, the ADMB command should automatically compile the code. However, if there is a header file for qfc_est.cpp, you'll have to include the header file, then compile with admb qfc_est.cpp S> On Oct 14, 2014, at 2:01 PM, Mark Maunder > wrote: From: Carolina Minte-Vera Sent: Tuesday, October 14, 2014 11:40 AM To: 'users at admb-project.org' Cc: Arni Magnusson (arnima at hafro.is) Subject: user-contributed libraries Dear Group and Dear Arni, I just installed the binary distribution admb-ide-101-win64.zip from ADMB at Google Code. I want to use some user-contributed libraries in my code such as qfc_est.cpp from http://admb-project.org/documentation/api/group__CONTRIB.html I could not find them in the admb-ide 10.1 I just installed. I would appreciate any hint on how to make them available for the ADMB version I am using. Just manually copy them? Thank you! Carolina Minte-Vera IATTC _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnima at hafro.is Tue Oct 14 15:19:04 2014 From: arnima at hafro.is (Arni Magnusson) Date: Tue, 14 Oct 2014 22:19:04 +0000 (UTC) Subject: [ADMB Users] user-contributed libraries In-Reply-To: References: Message-ID: Hi Caro, There is some garden work to do for the ADMB releases ... the newest ADMB-IDE is 10.1 and the newest ADMB for MinGW64 is also 10.1. To use ADMB 11.1 with MinGW, you can download and install the components individually: http://admb-project.googlecode.com/files/admb-11.1-windows-mingw-32bit.zip http://admb-project.org/tools/gcc/gcc472-win32.zip Once ADMB 11.1 is up and running, you can add the IDE components: ftp://ftp.gnu.org/gnu/emacs/windows/emacs-24.3-bin-i386.zip http://admb-project.org/tools/editors/emacs/admb.el http://admb-project.org/tools/admb-ide/eleven ADMB_HOME should point to a directory like c:/admb/admb111-gcc472-win32 HOME should point to a directory like c:/~ PATH should include ADMB, GCC, and Emacs admb.el should be in %HOME%/emacs/lisp/admb .emacs should be in %HOME% This is what I have on my old WinXP laptop and it runs smoothly with contributed libraries. Mainly a matter of unzipping and setting the environment variables. With the other hand on the fire extinguisher. Arni On Tue, 14 Oct 2014, Carolina Minte-Vera wrote: > Dear Group and Dear Arni, > > I just installed the binary distribution admb-ide-101-win64.zip from > ADMB at Google Code. I > want to use some user-contributed libraries in my code such as > qfc_est.cpp from > http://admb-project.org/documentation/api/group__CONTRIB.html I could > not find them in the admb-ide 10.1 I just installed. > > I would appreciate any hint on how to make them available for the ADMB > version I am using. Just manually copy them? > > Thank you! > Carolina Minte-Vera > IATTC > From davef at otter-rsch.com Fri Oct 17 06:27:17 2014 From: davef at otter-rsch.com (dave fournier) Date: Fri, 17 Oct 2014 06:27:17 -0700 Subject: [ADMB Users] building 64 bit mingw libraries for ADMB Message-ID: <54411935.7000908@otter-rsch.com> I recently had trouble with the windows installer for the 64 bit mingw C++ compiler for windows. Here is what worked for me to do a 64 bit compile ADMB source. I put everything in c:\ because I hate long directory names (with spaces). Get the source from http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/4.9.0/threads-posix/seh/x86_64-4.9.0-release-posix-seh-rt_v3-rev2.7z In the directory c:\ extract the source with 7zip you will get the files in c:\mingw64 Add c:\admb;c:\mingw64\bin to your PATH environment string. (Done with control panel.) Download ADMB source from http://admb-project.googlecode.com/files/admb-11.1.zip Again in c:\ unzip the code. You will have a directory admb. Change to that directory. Now go down one directory to src Add the line MINGW_HOME=c:\mingw64 to the file GNUmakefile Go up one directory to admb again. Type mingw32-make.exe This will build the admb 64 bit libraries. 1,1 Top From thomas.wainwright at noaa.gov Fri Oct 17 16:34:43 2014 From: thomas.wainwright at noaa.gov (Tom Wainwright) Date: Fri, 17 Oct 2014 16:34:43 -0700 Subject: [ADMB Users] RE model passing by minimum objective value? Message-ID: This may be a Friday-afternoon muddle-headed message about a well-known phenomenon, but I'm puzzled by some behavior in the newton-raphson iterations in RE models. I'm implementing a non-linear state-space model, and noticed that the "f=" values scrolling by on the screen were sometimes lower than what was showing up in my report files. It puzzled me enough to test a simple example, and the same thing happens: The n-r process reports intermediate f's that are lower than the final value. This may not be a big deal, but it seems odd that it passes by fits that are "better" than that to which it finally converges. To demonstrate, I used the RE version of the simple.tpl model (ADMB-re manual section 2.3). I've attached my version of the .tpl and the standard output (simple.out), the .dat file is in the distribution. When run, the standard output shows a minimum f of 20.07 (line 74 of simple.out), but the final statistics show a value of 21.74 (line 116), which is also the value in the .par file. I'm guessing this is because both values are within the convergence tolerances, so there's little difference in the result. But, its a bit disconcerting to see lower values discarded along the way. -- Tom Wainwright Research Fisheries Biologist Newport Research Station Northwest Fisheries Science Center National Oceanic and Atmospheric Administration thomas.wainwright at noaa.gov www.nwfsc.noaa.gov ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The contents of this message are mine personally and do not necessarily reflect any position of the Government or the National Oceanic and Atmospheric Administration. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: simple.tpl Type: application/vnd.groove-tool-template Size: 2379 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: simple.out Type: application/octet-stream Size: 8915 bytes Desc: not available URL: From davef at otter-rsch.com Fri Oct 17 20:42:23 2014 From: davef at otter-rsch.com (dave fournier) Date: Fri, 17 Oct 2014 20:42:23 -0700 Subject: [ADMB Users] RE model passing by minimum objective value? In-Reply-To: References: Message-ID: <5441E19F.9030503@otter-rsch.com> I could say Wow you are right I guess it never worked! Oh well too bad. or It is just reporting the results from the inner optimization, so it does not include the contribution from the Laplace approximation. From thomas.wainwright at noaa.gov Mon Oct 20 08:54:18 2014 From: thomas.wainwright at noaa.gov (Tom Wainwright) Date: Mon, 20 Oct 2014 08:54:18 -0700 Subject: [ADMB Users] RE model passing by minimum objective value? In-Reply-To: References: Message-ID: Thanks to both Dave and Jim - that clarifies the output and removes my confusion. I was misinterpreting the two lines printed out each iteration, one that gives "f =" and "max g =" and the next that gives "inner maxg =" and "Inner f =" - I was wrongly interpreting the first line as the values for the full model, not the inner. Tom On Fri, Oct 17, 2014 at 9:02 PM, James Thorson wrote: > Tom, > > I don't know it it'll be helpful, but attached please find a description > where we reverse-engineered ADMB-RE for use with Stock Synthesis. It might > clarify the difference between inner (joint likelihood) and outer (marginal > likelihood) optimization. > > cheers, > Jim > > On Fri, Oct 17, 2014 at 4:34 PM, Tom Wainwright < > thomas.wainwright at noaa.gov> wrote: > >> This may be a Friday-afternoon muddle-headed message about a well-known >> phenomenon, but I'm puzzled by some behavior in the newton-raphson >> iterations in RE models. >> >> I'm implementing a non-linear state-space model, and noticed that the >> "f=" values scrolling by on the screen were sometimes lower than what was >> showing up in my report files. It puzzled me enough to test a simple >> example, and the same thing happens: The n-r process reports intermediate >> f's that are lower than the final value. This may not be a big deal, but it >> seems odd that it passes by fits that are "better" than that to which it >> finally converges. >> >> To demonstrate, I used the RE version of the simple.tpl model (ADMB-re >> manual section 2.3). I've attached my version of the .tpl and the standard >> output (simple.out), the .dat file is in the distribution. >> >> When run, the standard output shows a minimum f of 20.07 (line 74 of >> simple.out), but the final statistics show a value of 21.74 (line 116), >> which is also the value in the .par file. >> >> I'm guessing this is because both values are within the convergence >> tolerances, so there's little difference in the result. But, its a bit >> disconcerting to see lower values discarded along the way. >> >> -- >> >> Tom Wainwright >> >> Research Fisheries Biologist >> >> Newport Research Station >> >> Northwest Fisheries Science Center >> >> National Oceanic and Atmospheric Administration >> >> thomas.wainwright at noaa.gov >> >> www.nwfsc.noaa.gov >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> The contents of this message are mine personally and do not >> necessarily reflect any position of the Government or the >> National Oceanic and Atmospheric Administration. >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> _______________________________________________ >> Users mailing list >> Users at admb-project.org >> http://lists.admb-project.org/mailman/listinfo/users >> >> > -- Tom Wainwright Research Fisheries Biologist Newport Research Station Northwest Fisheries Science Center National Oceanic and Atmospheric Administration www.nwfsc.noaa.gov ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The contents of this message are mine personally and do not necessarily reflect any position of the Government or the National Oceanic and Atmospheric Administration. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cminte at iattc.org Tue Oct 14 11:40:33 2014 From: cminte at iattc.org (Carolina Minte-Vera) Date: Tue, 14 Oct 2014 18:40:33 +0000 Subject: [ADMB Users] user-contributed libraries Message-ID: Dear Group and Dear Arni, I just installed the binary distribution admb-ide-101-win64.zip from ADMB at Google Code. I want to use some user-contributed libraries in my code such as qfc_est.cpp from http://admb-project.org/documentation/api/group__CONTRIB.html I could not find them in the admb-ide 10.1 I just installed. I would appreciate any hint on how to make them available for the ADMB version I am using. Just manually copy them? Thank you! Carolina Minte-Vera IATTC -------------- next part -------------- An HTML attachment was scrubbed... URL: From cminte at iattc.org Tue Oct 14 14:09:56 2014 From: cminte at iattc.org (Carolina Minte-Vera) Date: Tue, 14 Oct 2014 21:09:56 +0000 Subject: [ADMB Users] user-contributed libraries In-Reply-To: <1A113C62-90F3-4EAC-B45D-1AC54B6B5B2B@iphc.int> References: <1A113C62-90F3-4EAC-B45D-1AC54B6B5B2B@iphc.int> Message-ID: Hi Steve, this is what I thought, but I am putting the files in the directory "c:\admb\include\ " . Thanks a lot for the response, I try your idea too. Cheers, Carolina From: Steve Martell [mailto:SteveM at iphc.int] Sent: Tuesday, October 14, 2014 2:06 PM To: Mark Maunder Cc: users at admb-project.org; Carolina Minte-Vera Subject: Re: [ADMB Users] user-contributed libraries Quick and dirty, put a copy of qfc_est.cpp in your tpl directory. Then in the GLOBALS_SECTION use #include "qfc_est.cpp". When you compile the code, the ADMB command should automatically compile the code. However, if there is a header file for qfc_est.cpp, you'll have to include the header file, then compile with admb qfc_est.cpp S> On Oct 14, 2014, at 2:01 PM, Mark Maunder > wrote: From: Carolina Minte-Vera Sent: Tuesday, October 14, 2014 11:40 AM To: 'users at admb-project.org' Cc: Arni Magnusson (arnima at hafro.is) Subject: user-contributed libraries Dear Group and Dear Arni, I just installed the binary distribution admb-ide-101-win64.zip from ADMB at Google Code. I want to use some user-contributed libraries in my code such as qfc_est.cpp from http://admb-project.org/documentation/api/group__CONTRIB.html I could not find them in the admb-ide 10.1 I just installed. I would appreciate any hint on how to make them available for the ADMB version I am using. Just manually copy them? Thank you! Carolina Minte-Vera IATTC _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cminte at iattc.org Tue Oct 14 14:25:44 2014 From: cminte at iattc.org (Carolina Minte-Vera) Date: Tue, 14 Oct 2014 21:25:44 +0000 Subject: [ADMB Users] user-contributed libraries In-Reply-To: <1A113C62-90F3-4EAC-B45D-1AC54B6B5B2B@iphc.int> References: <1A113C62-90F3-4EAC-B45D-1AC54B6B5B2B@iphc.int> Message-ID: <4e4cceea-4d50-4aac-bd44-34f3e185b32a@MAIL1.lajolla.iattc.org> Hello Steve, thank you for your response. It worked! I copied the the .cpp and .h codes that I needed from the webpage (e.g. http://admb-project.org/documentation/api/qfc__sim_8cpp_source.html ) to the c:\admb\include directory and cleaned the line numbers in the code. Then my code compiles ok and includes what I need! Thanks. Carolina IATTC From: Steve Martell [mailto:SteveM at iphc.int] Sent: Tuesday, October 14, 2014 2:06 PM To: Mark Maunder Cc: users at admb-project.org; Carolina Minte-Vera Subject: Re: [ADMB Users] user-contributed libraries Quick and dirty, put a copy of qfc_est.cpp in your tpl directory. Then in the GLOBALS_SECTION use #include "qfc_est.cpp". When you compile the code, the ADMB command should automatically compile the code. However, if there is a header file for qfc_est.cpp, you'll have to include the header file, then compile with admb qfc_est.cpp S> On Oct 14, 2014, at 2:01 PM, Mark Maunder > wrote: From: Carolina Minte-Vera Sent: Tuesday, October 14, 2014 11:40 AM To: 'users at admb-project.org' Cc: Arni Magnusson (arnima at hafro.is) Subject: user-contributed libraries Dear Group and Dear Arni, I just installed the binary distribution admb-ide-101-win64.zip from ADMB at Google Code. I want to use some user-contributed libraries in my code such as qfc_est.cpp from http://admb-project.org/documentation/api/group__CONTRIB.html I could not find them in the admb-ide 10.1 I just installed. I would appreciate any hint on how to make them available for the ADMB version I am using. Just manually copy them? Thank you! Carolina Minte-Vera IATTC _______________________________________________ Users mailing list Users at admb-project.org http://lists.admb-project.org/mailman/listinfo/users ________________________________ This internet e-mail message, and any files transmitted with it, contains confidential, privileged information that is intended only for the addressee. If you have received this e-mail message in error, please call us at (206) 634-1838 collect if necessary) and ask to speak to the message sender. Nothing in this e-mail or the act of transmitting it, is to be construed as a waiver of any rights or privileges enjoyed by the sender or the International Pacific Halibut Commission pursuant to the International Organizations Immunities Act, 22 U.S.C. Sec. 288 et seq. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jefflaake at gmail.com Mon Oct 20 08:36:24 2014 From: jefflaake at gmail.com (Jeff Laake) Date: Mon, 20 Oct 2014 08:36:24 -0700 Subject: [ADMB Users] Non-matching analytic and numeric derivatives In-Reply-To: <54381B39.4010306@otter-rsch.com> References: <54381B39.4010306@otter-rsch.com> Message-ID: I want to follow up on this subject to summarize further discussions. The problem I was having was NOT the choice of the increment to avoid log(0). Possibly Dave ran into this because I forgot to include the .pin file when I sent him the files. The problem was due to having -gbs too small. I'm guessing that if -gbs is too small the code can't properly compute the analytic derivatives and the code fails to make any progress, Once I increased -gbs the code worked properly and converged. That said, what Dave stated here isn't exactly what he did. You have to be careful with the size of the epsilon you add to avoid log(0). I followed literally what he said here and he and I came up with substantially different log-likelihood values (-74 units). You need to use a value that is very small relative to the actual quantity. In my case they are probabilities that can be very small because it is a 27 year capture history. As it turns out, Dave started with 1e-8 and then switched to 1e-12 in the second phase. He implemented that by creating a dummy parameter (kkludge) that simply forced a second phase but which had value 0 and didn't affect the resulting value. Without showing entire tpl file, this shows the changes he made which added kkludge (set to 0 in pin) and shows how eps was changed at second phase. init_number kkludge(2) ... (calls function to compute f) ... f+=100.*square(kkludge); ... ... (in function to compute f) double myeps=1.e-12; if (current_phase()==1) myeps=1.e-8; f-= log(pch+myeps); // sum log-likelihood log(pr(ch)) Note that currently -gbs is limited to about 2gb on windows machines even for the 64 bit version. This is something that Johnoel is working on presently. regards --jeff On Fri, Oct 10, 2014 at 10:45 AM, dave fournier wrote: > On 10/09/2014 07:32 AM, Jeff Laake wrote: > > This might be of interest to others so I cc's it to the list. > > In your code for the log-likelihood you have the line. > > f-= log(pch+0.000000000000000000000001); > > Now I figured that 0.000000000000000000000001 might be too small. > > What would happen is that is pch is zero you will get derivatives on the > order of > > 1.0/ (0.000000000000000000000001) > > and this could make the model numerically unstable at least during the > initial fitting stage. > > I modified this to > > f-= log(pch+1.e-8); > > and ran the model for a while and then used the values as initial values to > run the > model with your original code. It converged easily. > > There is nothing magical about 1.e-8. For some models you might need > something like 1.e-4 to start. > > Dave > > >> Dave- >> >> Per Johnoel's request I forwarded a zip containing an example in which >> the numeric and analytic derivatives don't match when I use -dd. The >> code fails because it thinks it should move but can't make any >> progress because it is at the min. >> >> --jeff >> >> > From jefflaake at gmail.com Mon Oct 20 14:05:34 2014 From: jefflaake at gmail.com (Jeff Laake) Date: Mon, 20 Oct 2014 14:05:34 -0700 Subject: [ADMB Users] Non-matching analytic and numeric derivatives Message-ID: > I want to follow up on this subject to summarize further discussions. > The problem I was having was NOT the choice of the increment to avoid > log(0). Possibly Dave ran into this because I forgot to include the > .pin file when I sent him the files. The problem was due to having > -gbs too small. I'm guessing that if -gbs is too small the code can't > properly compute the analytic derivatives and the code fails to make > any progress, Once I increased -gbs the code worked properly and > converged. That said, what Dave stated here isn't exactly what he > did. You have to be careful with the size of the epsilon you add to > avoid log(0). I followed literally what he said here and he and I came > up with substantially different log-likelihood values (-74 units). > You need to use a value that is very small relative to the actual > quantity. In my case they are probabilities that can be very small > because it is a 27 year capture history. > > As it turns out, Dave started with 1e-8 and then switched to 1e-12 in > the second phase. He implemented that by creating a dummy parameter > (kkludge) that simply forced a second phase but which had value 0 and > didn't affect the resulting value. Without showing entire tpl file, > this shows the changes he made which added kkludge (set to 0 in pin) > and shows how eps was changed at second phase. > > init_number kkludge(2) > ... (calls function to compute f) > ... > f+=100.*square(kkludge); > ... > ... (in function to compute f) > double myeps=1.e-12; > if (current_phase()==1) > myeps=1.e-8; > f-= log(pch+myeps); // sum > log-likelihood log(pr(ch)) > > > Note that currently -gbs is limited to about 2gb on windows machines > even for the 64 bit version. This is something that Johnoel is > working on presently. > > regards --jeff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davef at otter-rsch.com Tue Oct 21 06:58:49 2014 From: davef at otter-rsch.com (dave fournier) Date: Tue, 21 Oct 2014 06:58:49 -0700 Subject: [ADMB Users] Non-matching analytic and numeric derivatives In-Reply-To: References: Message-ID: <54466699.5080209@otter-rsch.com> A few issues. I did not intend to present a "final solution" to your problem. What I did intend to do was to illustrate how overall stability of the model could be improved by modifying the objective function in phases. Then I hoped that you and others could incorporate the techniques into their bag of tricks. So for example one could use 3 phases and add an even smaller offset in the third phase. As for buffer sizes there are two issues. One is specifying the offset into a >2GB buffer. That has to be a type > 32 bits. In 64 bit windows it seems that a long int is only 32 bits and mingw64 has followed this for windows compatibility. A long long int will do the job but there are other types which can be used. I always forget what they are. Then there is the question of writing the buffer to the disk when it is full. I used a low level unbuffered write function, assuming at the time that it would be the fastest method. However the amount that can be written in one write operation is limited. On my Linux system it appears to be 2GB. So one can either split up the write operations or switch to higher level fwrites. For the model in question here I used a buffer > 3.5 GB which was large enough that nothing needed to be written to the disk. This works because in 64 bit Linux a long int is 64 bits. From davef at otter-rsch.com Tue Oct 21 19:44:08 2014 From: davef at otter-rsch.com (dave fournier) Date: Tue, 21 Oct 2014 19:44:08 -0700 Subject: [ADMB Users] [R-sig-ME] MCMC fitting in glmmADMB In-Reply-To: <1499133270.34898.1413928121734.open-xchange@patina.store> References: <5445C52A.9080801@otter-rsch.com> <1499133270.34898.1413928121734.open-xchange@patina.store> Message-ID: <544719F8.90404@otter-rsch.com> On 10/21/2014 02:48 PM, - - wrote: Hi, That sounds like a bug maybe. Maybe a memory leak. Don't remember exactly how it is implemented, but I don't think the memory use of any of the MCMC stuff should grow over time like that. If you send me what I need to run your model I'll take a look at it. Dave > Hi Dave, > > Sorry, I guess I didn?t explain my problem correctly in my original > request due to translation issues. It is the RAM and not the hard > drive space that is reaching its capacity. After running the MCMC > chain for 24 hours (approximately 2000 iterations for the model of my > own data) the allocated memory is already about 6 GB. > > I will send a follow up to the mailing list to clarify this. > > Best wishes, > Maren > > e -------------- next part -------------- An HTML attachment was scrubbed... URL: From bbolker at gmail.com Wed Oct 22 06:08:45 2014 From: bbolker at gmail.com (Ben Bolker) Date: Wed, 22 Oct 2014 09:08:45 -0400 Subject: [ADMB Users] [R-sig-ME] MCMC fitting in glmmADMB In-Reply-To: <544719F8.90404@otter-rsch.com> References: <5445C52A.9080801@otter-rsch.com> <1499133270.34898.1413928121734.open-xchange@patina.store> <544719F8.90404@otter-rsch.com> Message-ID: <5447AC5D.9070506@gmail.com> As a reminder, ?admbControl in the glmmADMB documentation says: if you want to try running the ADMB model outside of R, use ?run=FALSE? and set ?save.dir? in the ?glmmADMB? arguments. This will result in a saved directory containing the ?glmmadmb? executable (on systems other than Windows) as well as ?glmmadmb.pin? and ?glmmadmb.dat? files. You can then run the ?glmmadmb? executable with appropriate command-line arguments in a command shell or terminal window. If you run the same R command again (i.e. still with ?run=FALSE? and with the same ?save.dir?) ?glmmADMB? will find any output files that have been produced and read them into a ?glmmadmb? object. You can also try sending the ?glmmadmb.pin? and ?glmmadmb.dat? files to an ADMB expert for diagnosis. On 14-10-21 10:44 PM, dave fournier wrote: > On 10/21/2014 02:48 PM, - - wrote: > > Hi, > > That sounds like a bug maybe. Maybe a memory leak. Don't remember > exactly how it is implemented, but > I don't think the memory use of any of the MCMC stuff should grow over > time like that. If you send me > what I need to run your model I'll take a look at it. > > Dave > > >> Hi Dave, >> >> Sorry, I guess I didn?t explain my problem correctly in my original >> request due to translation issues. It is the RAM and not the hard >> drive space that is reaching its capacity. After running the MCMC >> chain for 24 hours (approximately 2000 iterations for the model of my >> own data) the allocated memory is already about 6 GB. >> >> I will send a follow up to the mailing list to clarify this. >> >> Best wishes, >> Maren >> >> e > > > > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > From davef at otter-rsch.com Wed Oct 22 18:29:58 2014 From: davef at otter-rsch.com (dave fournier) Date: Wed, 22 Oct 2014 18:29:58 -0700 Subject: [ADMB Users] memory leak In-Reply-To: <5448565D.5010205@gmail.com> References: <544846A5.1020201@otter-rsch.com> <5448565D.5010205@gmail.com> Message-ID: <54485A16.6050502@otter-rsch.com> On 10/22/2014 06:14 PM, Ben Bolker wrote: > On 14-10-22 08:07 PM, dave fournier wrote: >> The memory leak had nothing to do with glmmadmb as it turned out. It was >> a general admb thing which I fixed and posted >> to the list but it never showed up. By tomorrow I'll have forgotten what >> it was. > OK. Any chance of finding it and sending it to the list ... ? > > thanks for letting me know > > Ben > > -------------- next part -------------- A non-text attachment was scrubbed... Name: df1b2lp8.cpp Type: text/x-c++src Size: 30486 bytes Desc: not available URL: From davef at otter-rsch.com Fri Oct 24 10:30:32 2014 From: davef at otter-rsch.com (dave fournier) Date: Fri, 24 Oct 2014 10:30:32 -0700 Subject: [ADMB Users] fixed memory leak (hopefully) In-Reply-To: <544A3836.6040101@avitec-research.de> References: <5447E518.9020705@otter-rsch.com> <544A3836.6040101@avitec-research.de> Message-ID: <544A8CB8.9030101@otter-rsch.com> On 10/24/2014 04:29 AM, maren wrote: > > Hi Dave, > Yes there still seems to be a problem. the new one appears to be subtler than the original. Are you aware that you can do a smaller number of mcmc steps save the results and then restart the mcmc. that might be the way to go until this leak is identified. Dave > I have been running my model again for roughly 24 hours with the > updated version of glmmADMB that you sent to me. The allocated memory > might be a bit smaller but reached again already 5.5 GB. However, I > started 10000 iterations this time and not 2000 and cannot tell how > far the MCMC ran up to now (can I check that somehow?). Therefore, I > am wondering if there might be another memory leak. Or do I, for > example, also need to update the package R2admb in case the problem > was ADMB related? > > Best wishes, > > Maren > > > > > Am 22.10.2014 19:10, schrieb dave fournier: >> Hi, >> >> Here is a 64 bit windows version of glmmadmb with the fix I put in. >> YOu should unzip it >> and replace the glmmadmb.exe in your R dist and give it a try. >> >> Cheers, >> >> Dave > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davef at otter-rsch.com Sun Oct 26 10:15:29 2014 From: davef at otter-rsch.com (dave fournier) Date: Sun, 26 Oct 2014 10:15:29 -0700 Subject: [ADMB Users] fix for memory leak in glmmadmb Message-ID: <544D2C31.5090008@otter-rsch.com> This change to df1b2fnl.cpp deal with most of the memory leaks in glmmadmb. -------------- next part -------------- A non-text attachment was scrubbed... Name: df1b2fnl.cpp Type: text/x-c++src Size: 20389 bytes Desc: not available URL: From davef at otter-rsch.com Sun Oct 26 12:20:05 2014 From: davef at otter-rsch.com (dave fournier) Date: Sun, 26 Oct 2014 12:20:05 -0700 Subject: [ADMB Users] fix for memory leak in glmmadmb In-Reply-To: References: <544D2C31.5090008@otter-rsch.com> Message-ID: <544D4965.20600@otter-rsch.com> On 10/26/2014 11:40 AM, Johnoel Ancheta wrote: I found out there are a number of changes. I will compile it all under windows and linux and test both out and send it to you. > Thanks Dave, I'll add them in. > > On Sunday, October 26, 2014, dave fournier > wrote: > > This change to df1b2fnl.cpp deal with most of the memory leaks in > glmmadmb. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davef at otter-rsch.com Wed Oct 22 09:06:45 2014 From: davef at otter-rsch.com (dave fournier) Date: Wed, 22 Oct 2014 09:06:45 -0700 Subject: [ADMB Users] [R-sig-ME] MCMC fitting in glmmADMB In-Reply-To: <54479577.1030303@avitec-research.de> References: <5445C52A.9080801@otter-rsch.com> <1499133270.34898.1413928121734.open-xchange@patina.store> <544719F8.90404@otter-rsch.com> <54479577.1030303@avitec-research.de> Message-ID: <5447D615.7090207@otter-rsch.com> On 10/22/2014 04:31 AM, maren wrote: I found two memory leaks in df1b2lp8.cpp which were causing glmmadmb to run out of memory when doing mcmc. They would also affect other aspects of the analysis. The offending pointers are block_diagonal_re_list block_diagonal_fe_list In the code they should be checked to see if they are non zero and deleted if they are. code is like that below. // check if already allocated if (block_diagonal_re_list) { delete block_diagonal_re_list; block_diagonal_re_list=0; } block_diagonal_re_list = new imatrix(1,num_separable_calls, 1,itmp); if (block_diagonal_re_list ==0) { cerr << "error_allocating imatrix" << endl; ad_exit(1); } // check if already allocated if (block_diagonal_fe_list) { delete block_diagonal_fe_list; block_diagonal_fe_list=0; } block_diagonal_fe_list = new imatrix(1,num_separable_calls, 1,itmpf); if (block_diagonal_fe_list ==0) { cerr << "error_allocating imatrix" << endl; ad_exit(1); } Thanks to valgrind for making this easy to find. So we need to get this into glmmadmb somehow. > Hi Dave, > > Thank you very much for offering to look into the problem. I attached > a shortened version of my data set (the first 2000 lines out of > 11195). I tried it out and the model is in principal running with this > shortened version. But please let me know if you need a larger part of > the data set. > > Technically the data set is not owned by us. Therefore, please don?t > distribute the data set further (without prior permission). For the > same reason I made all the variable names general. I assumed that the > details of what exactly is analysed here are not important for the > problem. But if you need more details about the study or the variables > please don?t hesitate to ask. > > I attached the R-code to run the model below. In case it is of > importance: I am using the R version 3.1.0 and the glmmADMB version 0.8.0. > > Thanks! > > Maren > > #-----------------------------------------------------------------------------------------------------------------------# > > # Here is the R-code: > > library(glmmADMB) > > Data <- read.csv2("Rebke_Data_sample.csv", header=TRUE, > colClasses="character") > > Data$Response<- as.numeric(Data$Response) > > Data$Fixed1<- as.factor(Data$Fixed1) > > Data$Fixed1 <- relevel(Data$Fixed1, ref="Off") > > Data$Fixed2<- as.factor(Data$Fixed2) > > Data$Fixed2 <- relevel(Data$Fixed2, ref="1") > > Data$Fixed3<- as.factor(Data$Fixed3) > > Data$Random1<- as.factor(Data$Random1) > > Data$Random2<- as.factor(Data$Random2) > > Data$Random3<- as.factor(Data$Random3) > > Data$Random4<- as.factor(Data$Random4) > > Data$Offset<- as.numeric(Data$Offset) > > nsim <- 2000 > > thin <- 10 > > Model <- glmmadmb(Response ~ Fixed1 + Fixed2 + Fixed3 + (1|Random1) + > (1|Random2) + (1|Random3) + (1|Random4) + offset(log(Offset)), > data=Data, family="poisson", zeroInflation=TRUE,mcmc=TRUE, > mcmc.opts=mcmcControl(mcmc=nsim, mcsave=thin)) > > #-----------------------------------------------------------------------------------------------------------------------# > > > > > Am 22.10.2014 04:44, schrieb dave fournier: >> On 10/21/2014 02:48 PM, - - wrote: >> >> Hi, >> >> That sounds like a bug maybe. Maybe a memory leak. Don't remember >> exactly how it is implemented, but >> I don't think the memory use of any of the MCMC stuff should grow >> over time like that. If you send me >> what I need to run your model I'll take a look at it. >> >> Dave >> >> >>> Hi Dave, >>> >>> Sorry, I guess I didn?t explain my problem correctly in my original >>> request due to translation issues. It is the RAM and not the hard >>> drive space that is reaching its capacity. After running the MCMC >>> chain for 24 hours (approximately 2000 iterations for the model of >>> my own data) the allocated memory is already about 6 GB. >>> >>> I will send a follow up to the mailing list to clarify this. >>> >>> Best wishes, >>> Maren >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From davef at otter-rsch.com Wed Oct 22 10:10:48 2014 From: davef at otter-rsch.com (dave fournier) Date: Wed, 22 Oct 2014 10:10:48 -0700 Subject: [ADMB Users] fixed memory leak (hopefully) Message-ID: <5447E518.9020705@otter-rsch.com> Hi, Here is a 64 bit windows version of glmmadmb with the fix I put in. YOu should unzip it and replace the glmmadmb.exe in your R dist and give it a try. Cheers, Dave -------------- next part -------------- A non-text attachment was scrubbed... Name: glmmadmb.zip Type: application/zip Size: 1007748 bytes Desc: not available URL: From johnoel at hawaii.edu Wed Oct 22 20:42:13 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Wed, 22 Oct 2014 17:42:13 -1000 Subject: [ADMB Users] memory leak In-Reply-To: <54485A16.6050502@otter-rsch.com> References: <544846A5.1020201@otter-rsch.com> <5448565D.5010205@gmail.com> <54485A16.6050502@otter-rsch.com> Message-ID: Thanks commited fix http://www.admb-project.org/redmine/projects/issues/repository/revisions/2494 On Wed, Oct 22, 2014 at 3:29 PM, dave fournier wrote: > On 10/22/2014 06:14 PM, Ben Bolker wrote: > >> On 14-10-22 08:07 PM, dave fournier wrote: >> >>> The memory leak had nothing to do with glmmadmb as it turned out. It was >>> a general admb thing which I fixed and posted >>> to the list but it never showed up. By tomorrow I'll have forgotten what >>> it was. >>> >> OK. Any chance of finding it and sending it to the list ... ? >> >> thanks for letting me know >> >> Ben >> >> >> > > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Sun Oct 26 11:40:02 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Sun, 26 Oct 2014 08:40:02 -1000 Subject: [ADMB Users] fix for memory leak in glmmadmb In-Reply-To: <544D2C31.5090008@otter-rsch.com> References: <544D2C31.5090008@otter-rsch.com> Message-ID: Thanks Dave, I'll add them in. On Sunday, October 26, 2014, dave fournier wrote: > This change to df1b2fnl.cpp deal with most of the memory leaks in glmmadmb. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Sun Oct 26 13:23:22 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Sun, 26 Oct 2014 10:23:22 -1000 Subject: [ADMB Users] fix for memory leak in glmmadmb In-Reply-To: <544D4965.20600@otter-rsch.com> References: <544D2C31.5090008@otter-rsch.com> <544D4965.20600@otter-rsch.com> Message-ID: Committed changes to df1b2fnl.cpp. See http://www.admb-project.org/redmine/projects/issues/repository/revisions/2505 I can add the rest when you are ready. Thanks, Johnoel On Sun, Oct 26, 2014 at 9:20 AM, dave fournier wrote: > On 10/26/2014 11:40 AM, Johnoel Ancheta wrote: > > > I found out there are a number of changes. I will compile it all under > windows and linux and > test both out and send it to you. > > > > Thanks Dave, I'll add them in. > > On Sunday, October 26, 2014, dave fournier wrote: > >> This change to df1b2fnl.cpp deal with most of the memory leaks in >> glmmadmb. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Mon Oct 27 11:02:40 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Mon, 27 Oct 2014 08:02:40 -1000 Subject: [ADMB Users] Non-matching analytic and numeric derivatives In-Reply-To: <54466699.5080209@otter-rsch.com> References: <54466699.5080209@otter-rsch.com> Message-ID: Hi Jeff, Sorry for delay. I now understand the -gbs issue with MinGW64 and ADMB. It turns out that the 64 bit MinGW compiler uses 32 bit posix function lseek instead of 64 bit. This causes overflow problems with using 64bit integer as a input into a smaller 32 bit integer for integers greater than the max value for 32 bit. To use 64 bit, the compiler needs the option _FILE_OFFSET_BITS=64 to be defined. This has been done and committed to the repository. Please download the most current installer, http://www.admb-project.org/buildbot/snapshots/admb-latest-windows8-mingw64.exe Uninstall the previous first. Then open the ADMB command prompt on the desktop, > admb cjsre > cjsre -nohess -gbs 4000000000 Thank you for bring the issue to our attention, Please let me know if it works, Johnoel On Tue, Oct 21, 2014 at 3:58 AM, dave fournier wrote: > A few issues. > > I did not intend to present a "final solution" to your > problem. > > What I did intend to do was to illustrate how overall > stability of the model could be improved by modifying > the objective function in phases. Then I hoped that > you and others could incorporate the techniques into > their bag of tricks. So for example one could use 3 > phases and add an even smaller offset in the third phase. > > As for buffer sizes there are two issues. One is > specifying the offset into a >2GB buffer. That has > to be a type > 32 bits. In 64 bit windows it seems that > a long int is only 32 bits and mingw64 has followed this > for windows compatibility. A long long int will do the > job but there are other types which can be used. I always > forget what they are. > > Then there is the question of writing the buffer to the > disk when it is full. I used a low level unbuffered > write function, assuming at the time that it would be > the fastest method. However the amount that can be > written in one write operation is limited. On my Linux > system it appears to be 2GB. So one can either split up > the write operations or switch to higher level fwrites. > For the model in question here I used a buffer > 3.5 GB > which was large enough that nothing needed to be written > to the disk. This works because in 64 bit Linux a long > int is 64 bits. > > > > > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maren.rebke at avitec-research.de Thu Oct 23 04:35:50 2014 From: maren.rebke at avitec-research.de (maren) Date: Thu, 23 Oct 2014 13:35:50 +0200 Subject: [ADMB Users] [R-sig-ME] MCMC fitting in glmmADMB In-Reply-To: <5447D615.7090207@otter-rsch.com> References: <5445C52A.9080801@otter-rsch.com> <1499133270.34898.1413928121734.open-xchange@patina.store> <544719F8.90404@otter-rsch.com> <54479577.1030303@avitec-research.de> <5447D615.7090207@otter-rsch.com> Message-ID: <5448E816.3050200@avitec-research.de> Hi Dave, Thank you very much for fixing the memory leak and sending the new version of glmmadmb! It seems to run fine. I only had to use save.dir in the first run in order to create the .dat and other files in the same folder as glmmadmb.exe. Otherwise I get the error message attached below. I will now try to run a longer MCMC chain and check the memory usage. Thanks again for taking care of this problem so quickly! Best wishes, Maren Error message: Error in glmmadmb(Response ~ Fixed1 + Fixed2 + Fixed3 + (1 | Random1) +: The function maximizer failed (couldn't find STD file) Troubleshooting steps include (1) run with 'save.dir' set and inspect output files; (2) change run parameters: see '?admbControl' In addition: Warning message: running command 'C:\WINDOWS\system32\cmd.exe /c "C:/Users/maren/Documents/R/win-library/3.1/glmmADMB/bin/windows64/glmmadmb.exe" -maxfn 500 -maxph 5 -noinit -shess' had status 1 Am 22.10.2014 18:06, schrieb dave fournier: > On 10/22/2014 04:31 AM, maren wrote: > > > I found two memory leaks in df1b2lp8.cpp which were causing > glmmadmb to run out of memory when doing mcmc. They would > also affect other aspects of the analysis. > > The offending pointers are > > block_diagonal_re_list > block_diagonal_fe_list > > In the code they should be checked to see if they are non zero and > deleted if they are. code is like that below. > > // check if already allocated > if (block_diagonal_re_list) > { > delete block_diagonal_re_list; > block_diagonal_re_list=0; > } > block_diagonal_re_list = new imatrix(1,num_separable_calls, > 1,itmp); > if (block_diagonal_re_list ==0) > { > cerr << "error_allocating imatrix" << endl; > ad_exit(1); > } > // check if already allocated > if (block_diagonal_fe_list) > { > delete block_diagonal_fe_list; > block_diagonal_fe_list=0; > } > block_diagonal_fe_list = new imatrix(1,num_separable_calls, > 1,itmpf); > > if (block_diagonal_fe_list ==0) > { > cerr << "error_allocating imatrix" << endl; > ad_exit(1); > } > > > Thanks to valgrind for making this easy to find. > > So we need to get this into glmmadmb somehow. > > > >> Hi Dave, >> >> Thank you very much for offering to look into the problem. I attached >> a shortened version of my data set (the first 2000 lines out of >> 11195). I tried it out and the model is in principal running with >> this shortened version. But please let me know if you need a larger >> part of the data set. >> >> Technically the data set is not owned by us. Therefore, please don?t >> distribute the data set further (without prior permission). For the >> same reason I made all the variable names general. I assumed that the >> details of what exactly is analysed here are not important for the >> problem. But if you need more details about the study or the >> variables please don?t hesitate to ask. >> >> I attached the R-code to run the model below. In case it is of >> importance: I am using the R version 3.1.0 and the glmmADMB version >> 0.8.0. >> >> Thanks! >> >> Maren >> >> #-----------------------------------------------------------------------------------------------------------------------# >> >> # Here is the R-code: >> >> library(glmmADMB) >> >> Data <- read.csv2("Rebke_Data_sample.csv", header=TRUE, >> colClasses="character") >> >> Data$Response<- as.numeric(Data$Response) >> >> Data$Fixed1<- as.factor(Data$Fixed1) >> >> Data$Fixed1 <- relevel(Data$Fixed1, ref="Off") >> >> Data$Fixed2<- as.factor(Data$Fixed2) >> >> Data$Fixed2 <- relevel(Data$Fixed2, ref="1") >> >> Data$Fixed3<- as.factor(Data$Fixed3) >> >> Data$Random1<- as.factor(Data$Random1) >> >> Data$Random2<- as.factor(Data$Random2) >> >> Data$Random3<- as.factor(Data$Random3) >> >> Data$Random4<- as.factor(Data$Random4) >> >> Data$Offset<- as.numeric(Data$Offset) >> >> nsim <- 2000 >> >> thin <- 10 >> >> Model <- glmmadmb(Response ~ Fixed1 + Fixed2 + Fixed3 + (1|Random1) + >> (1|Random2) + (1|Random3) + (1|Random4) + offset(log(Offset)), >> data=Data, family="poisson", zeroInflation=TRUE,mcmc=TRUE, >> mcmc.opts=mcmcControl(mcmc=nsim, mcsave=thin)) >> >> #-----------------------------------------------------------------------------------------------------------------------# >> >> >> >> >> Am 22.10.2014 04:44, schrieb dave fournier: >>> On 10/21/2014 02:48 PM, - - wrote: >>> >>> Hi, >>> >>> That sounds like a bug maybe. Maybe a memory leak. Don't remember >>> exactly how it is implemented, but >>> I don't think the memory use of any of the MCMC stuff should grow >>> over time like that. If you send me >>> what I need to run your model I'll take a look at it. >>> >>> Dave >>> >>> >>>> Hi Dave, >>>> >>>> Sorry, I guess I didn?t explain my problem correctly in my original >>>> request due to translation issues. It is the RAM and not the hard >>>> drive space that is reaching its capacity. After running the MCMC >>>> chain for 24 hours (approximately 2000 iterations for the model of >>>> my own data) the allocated memory is already about 6 GB. >>>> >>>> I will send a follow up to the mailing list to clarify this. >>>> >>>> Best wishes, >>>> Maren >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maren.rebke at avitec-research.de Fri Oct 24 04:29:58 2014 From: maren.rebke at avitec-research.de (maren) Date: Fri, 24 Oct 2014 13:29:58 +0200 Subject: [ADMB Users] fixed memory leak (hopefully) In-Reply-To: <5447E518.9020705@otter-rsch.com> References: <5447E518.9020705@otter-rsch.com> Message-ID: <544A3836.6040101@avitec-research.de> Hi Dave, I have been running my model again for roughly 24 hours with the updated version of glmmADMB that you sent to me. The allocated memory might be a bit smaller but reached again already 5.5 GB. However, I started 10000 iterations this time and not 2000 and cannot tell how far the MCMC ran up to now (can I check that somehow?). Therefore, I am wondering if there might be another memory leak. Or do I, for example, also need to update the package R2admb in case the problem was ADMB related? Best wishes, Maren Am 22.10.2014 19:10, schrieb dave fournier: > Hi, > > Here is a 64 bit windows version of glmmadmb with the fix I put in. > YOu should unzip it > and replace the glmmadmb.exe in your R dist and give it a try. > > Cheers, > > Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Mon Oct 27 13:38:10 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Mon, 27 Oct 2014 10:38:10 -1000 Subject: [ADMB Users] ADMB and MinGW-w64 users should update Message-ID: An issue and been found by Jeff Laake and corrected. See http://lists.admb-project.org/pipermail/users/2014-October/002701.html Users that need buffers greater than 2GB, please download the most recent http://www.admb-project.org/buildbot/snapshots/admb-latest-windows8-mingw64.exe Follow the installation procedure below http://www.admb-project.org/buildbot/documentation/QuickStartWindows.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From davef at otter-rsch.com Tue Oct 28 07:29:39 2014 From: davef at otter-rsch.com (dave fournier) Date: Tue, 28 Oct 2014 07:29:39 -0700 Subject: [ADMB Users] ADMB and MinGW-w64 users should update In-Reply-To: References: Message-ID: <544FA853.1050801@otter-rsch.com> As I recall there were two issues involved with having a buffer size > 2GB. First was to use an 8 byte integer type for the offset. The second issue was that if the buffer gets full it is saved to disk with a low level unbuffered write. This write could not write more than 2GB for me. Of course if the buffer is large enough so that it never gets full that is not a problem. From davef at otter-rsch.com Tue Oct 28 07:42:44 2014 From: davef at otter-rsch.com (dave fournier) Date: Tue, 28 Oct 2014 07:42:44 -0700 Subject: [ADMB Users] fixed memory leak (hopefully) In-Reply-To: <544F5AA1.7090908@avitec-research.de> References: <5447E518.9020705@otter-rsch.com> <544A3836.6040101@avitec-research.de> <544D5C88.3050201@otter-rsch.com> <544F5AA1.7090908@avitec-research.de> Message-ID: <544FAB64.1040300@otter-rsch.com> On 10/28/2014 01:58 AM, maren wrote: Good to hear. I hope we can get glmmadmb rebuilt with these changes. Dave > Hi Dave, > > I just want to let you know that the new version of glmmADMB is > performing well in a long MCMC run. The allocated memory doesn?t reach > such large sizes anymore. > > Thanks again for fixing the memory leaks! > > Best wishes, > > Maren > > > > > Am 26.10.2014 21:41, schrieb dave fournier: >> On 10/24/2014 04:29 AM, maren wrote: >> >> Hi, >> >> >> This should be much better. >> >> Dave >> >> >>> Hi Dave, >>> >>> I have been running my model again for roughly 24 hours with the >>> updated version of glmmADMB that you sent to me. The allocated >>> memory might be a bit smaller but reached again already 5.5 GB. >>> However, I started 10000 iterations this time and not 2000 and >>> cannot tell how far the MCMC ran up to now (can I check that >>> somehow?). Therefore, I am wondering if there might be another >>> memory leak. Or do I, for example, also need to update the package >>> R2admb in case the problem was ADMB related? >>> >>> Best wishes, >>> >>> Maren >>> >>> >>> >>> >>> Am 22.10.2014 19:10, schrieb dave fournier: >>>> Hi, >>>> >>>> Here is a 64 bit windows version of glmmadmb with the fix I put in. >>>> YOu should unzip it >>>> and replace the glmmadmb.exe in your R dist and give it a try. >>>> >>>> Cheers, >>>> >>>> Dave >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnoel at hawaii.edu Tue Oct 28 13:38:57 2014 From: johnoel at hawaii.edu (Johnoel Ancheta) Date: Tue, 28 Oct 2014 20:38:57 -0000 Subject: [ADMB Users] ADMB and MinGW-w64 users should update In-Reply-To: <544FA853.1050801@otter-rsch.com> References: <544FA853.1050801@otter-rsch.com> Message-ID: The first issue was corrected. See http://www.admb-project.org/redmine/projects/issues/repository/revisions/2498 The second issue, I will need to look into. MinGW-w64 does support large files, but I will need to test this. On Tue, Oct 28, 2014 at 4:29 AM, dave fournier wrote: > As I recall there were two issues involved with having a buffer size > > 2GB. First was > to use an 8 byte integer type for the offset. > > The second issue was that if the buffer gets full it is saved to disk with > a low level unbuffered > write. This write could not write more than 2GB for me. Of course if the > buffer is large > enough so that it never gets full that is not a problem. > _______________________________________________ > Users mailing list > Users at admb-project.org > http://lists.admb-project.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: