[Developers] profiling ADMB model with gprof

John Sibert sibert at hawaii.edu
Wed Nov 17 09:07:14 PST 2010


This may be a clue. Ian's gprof output contains the entry
> NA      9.1      dvector::operator()(int) 
That suggest to me that something got compiled without the -DOPT_LIB 
flag. I cannot find such an entry in the gprof output from models built 
using v9 (I haven't check v10 yet). Where does the NA come from?

John

On 11/16/2010 12:31 PM, Ian Taylor wrote:
>  On 11/15/2010 10:52 AM, dave fournier wrote:
>> I suspect the difference could be in writing to temporary files.
> I didn't see any evidence of temporary files getting written. I did 
> notice that the compile and link commands for v10beta didn't have a 
> -m64 command, but adding it in didn't seem to make any difference. 
> Indeed, the only thing that seems to make a difference on speed was 
> whether I was linking to the new libraries or the old.
>
> Here's some output from an additional test that I just conducted with 
> the "catage" example included with ADMB, where I modified the TPL to 
> provide iternations and run times and I modified the data file to 
> include 200 years and 40 ages so that it would run slower (files are 
> attached). The commands below are identical with the exception of 
> whether they point to the v9 or v10 directories. As you can see from 
> the last section, the temporary files were never used and the run 
> times were 17 seconds in v9 and 50 seconds in v10. (This was all done 
> with GCC 4.1.2 on 64 bit Red Hat Enterprise Linux Server release 5.5 
> with 3 GHz processor and 32 GB memory.) The results continue to be 
> identical, which is an important first step. However, I think I'll 
> stick with the old code until I hear news that somebody else has 
> achieved similar performance between versions 9 and 10.
> -Ian
>
> [catage2]$ # convert
> [catage2]$ tpl2cpp catage2_v9
> [catage2]$ tpl2cpp catage2_v10
> [catage2]$
> [catage2]$ # compile
> [catage2]$ g++ -c -m64  -O3 -Wno-deprecated -D__GNUDOS__  -Dlinux 
> -DOPT_LIB -DUSE_LAPLACE -fpermissive -I. 
> -I/home/itaylor/h_itaylor/admb/v9/include  catage2_v9.cpp
> [catage2]$ g++ -c -m64  -O3 -Wno-deprecated -D__GNUDOS__  -Dlinux 
> -DOPT_LIB -DUSE_LAPLACE -fpermissive -I. 
> -I/home/itaylor/h_itaylor/admb/v10/include catage2_v10.cpp
> [catage2]$
> [catage2]$ # link
> [catage2]$ g++ -m64   -L/home/itaylor/h_itaylor/admb/v9/lib  
> catage2_v9.o  -ldf1b2o -ladmod -ladt -lado -ldf1b2o -ladmod -ladt 
> -lado -o catage2_v9
> [catage2]$ g++ -m64   -L/home/itaylor/h_itaylor/admb/v10/lib 
> catage2_v10.o -ldf1b2o -ladmod -ladt -lado -ldf1b2o -ladmod -ladt 
> -lado -o catage2_v10
> [catage2]$
> [catage2]$ # run
> [catage2]$ ./catage2_v9  -gbs 1000000000 -cbs 1000000000 -nox > 
> run_notes_v9.txt
> [catage2]$ ./catage2_v10 -gbs 1000000000 -cbs 1000000000 -nox > 
> run_notes_v10.txt
> [catage2]$
> [catage2]$ # check for temporary file useage
> [catage2]$ cat catage2_v9.log
> size of file gradfil1.tmp = 0
> size of file gradfil2.tmp = 0
> size of file varssave.tmp = 0
> size of file cmpdiff.tmp = 0
> [catage2]$ cat catage2_v10.log
> size of file gradfil1.tmp = 0
> size of file gradfil2.tmp = 0
> size of file varssave.tmp = 0
> size of file cmpdiff.tmp = 0
> [catage2]$
> [catage2]$ # run times
> [catage2]$ head -2 catage2_v9.rep
> 1351 # N iternations
> 17 # runtime (sec)
> [catage2]$ head -2 catage2_v10.rep
> 1351 # N iternations
> 50 # runtime (sec)
>
>
>>
>>> The suggestion of trying gprof is something that I'd been meaning to 
>>> do in the past, and now there was an incentive.
>>> I changed the subject line of this email to better reflect that this 
>>> is now the topic.
>>>
>>> I haven't seen any instructions on using gprof for ADMB, so I posted 
>>> the steps that eventually succeeded at 
>>> http://admb-project.org/community/tutorials-and-examples/how-to-use-gprof-the-gnu-profiler. 
>>> If it turns out that these options were already built into the 
>>> existing scripts or if there's an easier way, let me know so I can 
>>> update the instructions on the web. (If the instructions are any 
>>> good, then maybe they could be built upon by Hans, Johnoel, Jim I. 
>>> and Ray Conser, who were assigned the lower priority topic of "code 
>>> tuning" 
>>> <http://admb-project.org/community/admb-meeting-march-29-31/ADMBMeetingReportv3.pdf#page=4> 
>>> at the past ADMB meeting).
>>>
>>> I haven't really digested the results yet, because there are a lot 
>>> of differences between the profiles I got for ADMB versions 9 and 
>>> 10. However, here's a sample of the function calls that took up the 
>>> most time, matched up with the times (in seconds) from version 9 
>>> (compiled from the identical TPL file and identical inputs):
>>>
>>> v9.secs v10.secs function.name
>>> 2.62    14.8     
>>> dvector::save_dvector_derivatives(dvar_vector_position const&) const
>>> 5.72    9.43     DF_FILE::fread(double const&)
>>> NA      9.1      dvector::operator()(int)
>>> 0.61    6.55     gradient_structure::jacobcalc(int, 
>>> std::basic_ofstream<char, std::char_traits<char> > const&)
>>> 1.05    6.21     
>>> restore_dvar_vector_derivatives(dvar_vector_position const&)
>>> 0.52    5.59     gradcalc(int, dvector const&)
>>> NA      5.08     dvector::elem(int)
>>> 1.66    4.33     DF_FILE::fwrite(double)
>>> 1.96    4.21     restore_dvar_vector_value(dvar_vector_position const&)
>>> NA      3.86     dvar_vector_position::indexmin() const
>>>
>>> These functions that are taking the most time seem to be deep in 
>>> ADMB (or AUTODIF) and not stuff that's easy for a user to work 
>>> around. If I'm not the only one finding version 10 to be slower than 
>>> version 9, then I'm guessing that somebody else is better equipped 
>>> to suggest what to do about it.
>>> -Ian
>>>
>>> On 11/12/2010 6:53 AM, dave fournier wrote:
>>>> Ian Taylor wrote:
>>>>
>>>> You could compile it for profiling and find out where the time is 
>>>> spent
>>>> with gprof.
>>>>> After sending the yesterday, I caught the missing -O3. I tried 
>>>>> adding it back and also just doing the compiling with the old 
>>>>> script. But as long as I used the new libraries at the adlink 
>>>>> stage, it went slow. And by slow, I mean more than 8 minutes for a 
>>>>> model that previously took 2 1/2 minutes in ADMB version 9 if 
>>>>> optimized and just over 3 minutes in safe mode (which also 
>>>>> suggests that it's not just a matter of optimizing).
>>>>>
>>>>> I can try other models to see if the pattern I saw was consistent, 
>>>>> but I'm also curious if anyone else has compared the speeds 
>>>>> between versions. I don't know if this is in the documentation, 
>>>>> but run times can be quantified with code like the following.
>>>>> -Ian
>>>>>
>>>>>     TOP_OF_MAIN_SECTION
>>>>>       time(&start);
>>>>>
>>>>>     GLOBALS_SECTION
>>>>>       #include <time.h>
>>>>>       time_t start,finish;
>>>>>       double elapsed_time;
>>>>>
>>>>>     FINAL_SECTION
>>>>>       time(&finish);
>>>>>       elapsed_time = difftime(finish,start);
>>>>>
>>>>> On 11/12/2010 2:09 PM, Johnoel Ancheta wrote:
>>>>>> Opps... Sorry...
>>>>>>
>>>>>> I accidentally committed the Makefile without the -O3 option.
>>>>>>
>>>>>> I'll correct and commit the changes.
>>>>>>
>>>>>> Thanks Dave for catching that.
>>>>>>
>>>>>> Johnoel
>>>>>>
>>>>>> On 11/12/10 4:09 AM, dave fournier wrote:
>>>>>>> Ian Taylor wrote:
>>>>>>>
>>>>>>> Just a thought Re speed. I don't see a -O3 flag for gcc for 
>>>>>>> optimization.
>>>>>>>> Hi Johnoel and others,
>>>>>>>>
>>>>>>>> Compiling ADMB 10 Beta on with 64-bit linux worked perfectly 
>>>>>>>> for me with the files in the download.
>>>>>>>>
>>>>>>>> However, the new script you sent by email failed. From the 
>>>>>>>> commands below (fuller notes attached), it looks like for 
>>>>>>>> Ubuntu at least, it would be better to test for 64 vs 32 using 
>>>>>>>> uname -m rather than uname -p.
>>>>>>>>
>>>>>>>>     ~/admb/admb-10.0Beta $ uname -a
>>>>>>>>     Linux ian-laptop 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16
>>>>>>>>     19:52:42 UTC 2010 x86_64 GNU/Linux
>>>>>>>>     ~/admb/admb-10.0Beta $ uname -s
>>>>>>>>     Linux
>>>>>>>>     ~/admb/admb-10.0Beta $ uname -p
>>>>>>>>     unknown
>>>>>>>>     ~/admb/admb-10.0Beta $ uname -m
>>>>>>>>     x86_64
>>>>>>>>
>>>>>>>> Aside from the good news of the successful install, this 
>>>>>>>> version 10 Beta also produced identical results, which is also 
>>>>>>>> good news. However, it seems to be running slower than version 
>>>>>>>> 9, by a large margin, at least for a single model that I tried. 
>>>>>>>> Perhaps related to this, when I compiled using GCC in and ADMB 
>>>>>>>> version 9, the safe executable was about double the size of the 
>>>>>>>> optimized. Compiling with ADMB version 10, the file sizes are 
>>>>>>>> similar, and in between the old sizes. Perhaps the speed and 
>>>>>>>> file size differences are related to changes in the compile 
>>>>>>>> scripts? My old script was dated "23 May 2009".
>>>>>>>>
>>>>>>>> Finally, thank you, Johnoel, and all the others who've 
>>>>>>>> contributed to the developments that have gone into this new 
>>>>>>>> beta release.
>>>>>>>> -Ian
>>>>>>>>
>>>>>>>> On 11/10/2010 02:44 PM, Johnoel Ancheta wrote:
>>>>>>>>> Updated configure script is available.
>>>>>>>>>
>>>>>>>>> http://admb-project.googlecode.com/files/configure
>>>>>>>>>
>>>>>>>>> Download and replace the configure script in the admb-10.0Beta 
>>>>>>>>> distribution.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 11/10/10 11:09 AM, Ben Bolker wrote:
>>>>>>>>>>    (running under VMWare Fusion on Mac OS X.6, but I don't 
>>>>>>>>>> think that
>>>>>>>>>> matters)
>>>>>>>>>>
>>>>>>>>>>    It looks like configure is getting confused and thinking 
>>>>>>>>>> my system is
>>>>>>>>>> 64-bit ... ?
>>>>>>>>>>
>>>>>>>>>>> uname -a
>>>>>>>>>> Linux ubuntu-10 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 
>>>>>>>>>> 19:48:22 UTC
>>>>>>>>>> 2010 i686 GNU/Linux
>>>>>>>>>>
>>>>>>>>>> uname -m
>>>>>>>>>> i686  ## indicates 32-bit kernel
>>>>>>>>>>
>>>>>>>>>> $ gcc -v
>>>>>>>>>> Using built-in specs.
>>>>>>>>>> Target: i486-linux-gnu
>>>>>>>>>> Configured with: ../src/configure -v --with-pkgversion='Ubuntu
>>>>>>>>>> 4.4.3-4ubuntu5' 
>>>>>>>>>> --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
>>>>>>>>>> --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
>>>>>>>>>> --enable-shared --enable-multiarch --enable-linker-build-id
>>>>>>>>>> --with-system-zlib --libexecdir=/usr/lib 
>>>>>>>>>> --without-included-gettext
>>>>>>>>>> --enable-threads=posix 
>>>>>>>>>> --with-gxx-include-dir=/usr/include/c++/4.4
>>>>>>>>>> --program-suffix=-4.4 --enable-nls --enable-clocale=gnu
>>>>>>>>>> --enable-libstdcxx-debug --enable-plugin --enable-objc-gc
>>>>>>>>>> --enable-targets=all --disable-werror --with-arch-32=i486
>>>>>>>>>> --with-tune=generic --enable-checking=release 
>>>>>>>>>> --build=i486-linux-gnu
>>>>>>>>>> --host=i486-linux-gnu --target=i486-linux-gnu
>>>>>>>>>> Thread model: posix
>>>>>>>>>> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$ 
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> 
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> 
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> sudo 
>>>>>>>>>> ./configure
>>>>>>>>>> checking for gcc... gcc
>>>>>>>>>> checking whether the C compiler works... yes
>>>>>>>>>> checking for C compiler default output file name... a.out
>>>>>>>>>> checking for suffix of executables...
>>>>>>>>>> checking whether we are cross compiling... no
>>>>>>>>>> checking for suffix of object files... o
>>>>>>>>>> checking whether we are using the GNU C compiler... yes
>>>>>>>>>> checking whether gcc accepts -g... yes
>>>>>>>>>> checking for gcc option to accept ISO C89... none needed
>>>>>>>>>> checking for g++... g++
>>>>>>>>>> checking whether we are using the GNU C++ compiler... yes
>>>>>>>>>> checking whether g++ accepts -g... yes
>>>>>>>>>> checking how to run the C++ preprocessor... g++ -E
>>>>>>>>>> checking for grep that handles long lines and -e... /bin/grep
>>>>>>>>>> checking for egrep... /bin/grep -E
>>>>>>>>>> checking for ANSI C header files... yes
>>>>>>>>>> checking for sys/types.h... yes
>>>>>>>>>> checking for sys/stat.h... yes
>>>>>>>>>> checking for stdlib.h... yes
>>>>>>>>>> checking for string.h... yes
>>>>>>>>>> checking for memory.h... yes
>>>>>>>>>> checking for strings.h... yes
>>>>>>>>>> checking for inttypes.h... yes
>>>>>>>>>> checking for stdint.h... yes
>>>>>>>>>> checking for unistd.h... yes
>>>>>>>>>> checking iostream usability... yes
>>>>>>>>>> checking iostream presence... yes
>>>>>>>>>> checking for iostream... yes
>>>>>>>>>> checking for flex... flex
>>>>>>>>>> checking lex output file root... lex.yy
>>>>>>>>>> checking lex library... none needed
>>>>>>>>>> checking whether yytext is a pointer... no
>>>>>>>>>> checking for sed... yes
>>>>>>>>>> checking whether C++ compiler is 64-bit enabled... yes
>>>>>>>>>> checking for svnversion... yes
>>>>>>>>>> configure: creating ./config.status
>>>>>>>>>> config.status: creating GNUmakefile
>>>>>>>>>> config.status: creating admb_configure.h
>>>>>>>>>> config.status: creating scripts/bash/admb-bin
>>>>>>>>>>
>>>>>>>>>> Configure completed for admb10.0Beta-linux-gcc4.4.3-64bit
>>>>>>>>>> To build ADMB, type 'make'
>>>>>>>>>> bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$ 
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> 
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> 
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> sudo 
>>>>>>>>>> make
>>>>>>>>>> CXX=g++ CXXFLAGS="-m64"
>>>>>>>>>> LIBPATH=../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects
>>>>>>>>>> DISK=../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist 
>>>>>>>>>> ADMB_CONFIGURE=1
>>>>>>>>>> make --directory=src --file=linux.mak
>>>>>>>>>> make[1]: Entering directory `/usr/local/src/admb-10.0Beta/src'
>>>>>>>>>> rm -rf ../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist/{bin,lib,include,docs,docs/manuals,examples} 
>>>>>>>>>>
>>>>>>>>>> mkdir -p 
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/linad99-olp
>>>>>>>>>> mkdir -p 
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/linad99-slp
>>>>>>>>>> mkdir -p 
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/nh99-olp
>>>>>>>>>> mkdir -p 
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/nh99-olp-stub
>>>>>>>>>> mkdir -p 
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/tools99-olp
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-slp 
>>>>>>>>>>
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp 
>>>>>>>>>>
>>>>>>>>>> make --directory=df1b2-separable CC=gcc CXXFLAGS="-c -m64 -Wall
>>>>>>>>>> -Wno-deprecated -DUSE_LAPLACE -fpermissive -I../df1b2-separable
>>>>>>>>>> -I../nh99 -I../linad99 -I../tools99 -D__SPDLL__ -D__GNUDOS__ 
>>>>>>>>>> -Dlinux
>>>>>>>>>> -DOPT_LIB"
>>>>>>>>>> LIBPATH=../../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp 
>>>>>>>>>>
>>>>>>>>>> DISKDIR=../../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist -f
>>>>>>>>>> optg32-rh8-laplace.mak disk
>>>>>>>>>> make[2]: Entering directory
>>>>>>>>>> `/usr/local/src/admb-10.0Beta/src/df1b2-separable'
>>>>>>>>>> g++ -c -m64 -Wall -Wno-deprecated -DUSE_LAPLACE -fpermissive
>>>>>>>>>> -I../df1b2-separable -I../nh99 -I../linad99 -I../tools99 
>>>>>>>>>> -D__SPDLL__
>>>>>>>>>> -D__GNUDOS__ -Dlinux -DOPT_LIB  df1b2impspf.cpp -o
>>>>>>>>>> ../../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp/df1b2impspf.obj 
>>>>>>>>>>
>>>>>>>>>> In file included from /usr/include/features.h:378,
>>>>>>>>>>                   from /usr/include/unistd.h:26,
>>>>>>>>>>                   from ../linad99/fvar.hpp:173,
>>>>>>>>>>                   from ../nh99/admodel.h:69,
>>>>>>>>>>                   from df1b2impspf.cpp:8:
>>>>>>>>>> /usr/include/gnu/stubs.h:9:27: error: gnu/stubs-64.h: No such 
>>>>>>>>>> file or
>>>>>>>>>> directory
>>>>>>>>>> df1b2impspf.cpp: In function ‘double
>>>>>>>>>> calculate_importance_sample_shess(const dvector&, const 
>>>>>>>>>> dvector&, const
>>>>>>>>>> dmatrix&, const dvector&, const dvector&, const dmatrix&,
>>>>>>>>>> function_minimizer*)’:
>>>>>>>>>> df1b2impspf.cpp:20: warning: unused variable ‘Hessadjoint’
>>>>>>>>>> df1b2impspf.cpp:44: warning: unused variable ‘onvar’
>>>>>>>>>> df1b2impspf.cpp:49: warning: unused variable ‘j’
>>>>>>>>>> make[2]: *** [df1b2impspf.obj] Error 1
>>>>>>>>>> make[2]: Leaving directory
>>>>>>>>>> `/usr/local/src/admb-10.0Beta/src/df1b2-separable'
>>>>>>>>>> make[1]: *** [dist] Error 2
>>>>>>>>>> make[1]: Leaving directory `/usr/local/src/admb-10.0Beta/src'
>>>>>>>>>> make: *** [all] Error 2
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Developers mailing list
>>>>>>>>>> Developers at admb-project.org 
>>>>>>>>>> <mailto:Developers at admb-project.org> 
>>>>>>>>>> <mailto:Developers at admb-project.org> 
>>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Developers mailing list
>>>>>>>>> Developers at admb-project.org 
>>>>>>>>> <mailto:Developers at admb-project.org> 
>>>>>>>>> <mailto:Developers at admb-project.org> 
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>> ------------------------------------------------------------------------ 
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Developers mailing list
>>>>>>>> Developers at admb-project.org 
>>>>>>>> <mailto:Developers at admb-project.org> 
>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>
>>>>>>
>>>>
>>> ------------------------------------------------------------------------ 
>>>
>>>
>>> _______________________________________________
>>> Developers mailing list
>>> Developers at admb-project.org
>>> http://lists.admb-project.org/mailman/listinfo/developers
>>
>
> _______________________________________________
> Developers mailing list
> Developers at admb-project.org
> http://lists.admb-project.org/mailman/listinfo/developers
>    

-- 
John Sibert
Emeritus Researcher, SOEST
University of Hawaii at Manoa

Visit the ADMB project http://admb-project.org/




More information about the Developers mailing list