[Developers] profiling ADMB model with gprof
John Sibert
sibert at hawaii.edu
Wed Nov 17 09:07:14 PST 2010
This may be a clue. Ian's gprof output contains the entry
> NA 9.1 dvector::operator()(int)
That suggest to me that something got compiled without the -DOPT_LIB
flag. I cannot find such an entry in the gprof output from models built
using v9 (I haven't check v10 yet). Where does the NA come from?
John
On 11/16/2010 12:31 PM, Ian Taylor wrote:
> On 11/15/2010 10:52 AM, dave fournier wrote:
>> I suspect the difference could be in writing to temporary files.
> I didn't see any evidence of temporary files getting written. I did
> notice that the compile and link commands for v10beta didn't have a
> -m64 command, but adding it in didn't seem to make any difference.
> Indeed, the only thing that seems to make a difference on speed was
> whether I was linking to the new libraries or the old.
>
> Here's some output from an additional test that I just conducted with
> the "catage" example included with ADMB, where I modified the TPL to
> provide iternations and run times and I modified the data file to
> include 200 years and 40 ages so that it would run slower (files are
> attached). The commands below are identical with the exception of
> whether they point to the v9 or v10 directories. As you can see from
> the last section, the temporary files were never used and the run
> times were 17 seconds in v9 and 50 seconds in v10. (This was all done
> with GCC 4.1.2 on 64 bit Red Hat Enterprise Linux Server release 5.5
> with 3 GHz processor and 32 GB memory.) The results continue to be
> identical, which is an important first step. However, I think I'll
> stick with the old code until I hear news that somebody else has
> achieved similar performance between versions 9 and 10.
> -Ian
>
> [catage2]$ # convert
> [catage2]$ tpl2cpp catage2_v9
> [catage2]$ tpl2cpp catage2_v10
> [catage2]$
> [catage2]$ # compile
> [catage2]$ g++ -c -m64 -O3 -Wno-deprecated -D__GNUDOS__ -Dlinux
> -DOPT_LIB -DUSE_LAPLACE -fpermissive -I.
> -I/home/itaylor/h_itaylor/admb/v9/include catage2_v9.cpp
> [catage2]$ g++ -c -m64 -O3 -Wno-deprecated -D__GNUDOS__ -Dlinux
> -DOPT_LIB -DUSE_LAPLACE -fpermissive -I.
> -I/home/itaylor/h_itaylor/admb/v10/include catage2_v10.cpp
> [catage2]$
> [catage2]$ # link
> [catage2]$ g++ -m64 -L/home/itaylor/h_itaylor/admb/v9/lib
> catage2_v9.o -ldf1b2o -ladmod -ladt -lado -ldf1b2o -ladmod -ladt
> -lado -o catage2_v9
> [catage2]$ g++ -m64 -L/home/itaylor/h_itaylor/admb/v10/lib
> catage2_v10.o -ldf1b2o -ladmod -ladt -lado -ldf1b2o -ladmod -ladt
> -lado -o catage2_v10
> [catage2]$
> [catage2]$ # run
> [catage2]$ ./catage2_v9 -gbs 1000000000 -cbs 1000000000 -nox >
> run_notes_v9.txt
> [catage2]$ ./catage2_v10 -gbs 1000000000 -cbs 1000000000 -nox >
> run_notes_v10.txt
> [catage2]$
> [catage2]$ # check for temporary file useage
> [catage2]$ cat catage2_v9.log
> size of file gradfil1.tmp = 0
> size of file gradfil2.tmp = 0
> size of file varssave.tmp = 0
> size of file cmpdiff.tmp = 0
> [catage2]$ cat catage2_v10.log
> size of file gradfil1.tmp = 0
> size of file gradfil2.tmp = 0
> size of file varssave.tmp = 0
> size of file cmpdiff.tmp = 0
> [catage2]$
> [catage2]$ # run times
> [catage2]$ head -2 catage2_v9.rep
> 1351 # N iternations
> 17 # runtime (sec)
> [catage2]$ head -2 catage2_v10.rep
> 1351 # N iternations
> 50 # runtime (sec)
>
>
>>
>>> The suggestion of trying gprof is something that I'd been meaning to
>>> do in the past, and now there was an incentive.
>>> I changed the subject line of this email to better reflect that this
>>> is now the topic.
>>>
>>> I haven't seen any instructions on using gprof for ADMB, so I posted
>>> the steps that eventually succeeded at
>>> http://admb-project.org/community/tutorials-and-examples/how-to-use-gprof-the-gnu-profiler.
>>> If it turns out that these options were already built into the
>>> existing scripts or if there's an easier way, let me know so I can
>>> update the instructions on the web. (If the instructions are any
>>> good, then maybe they could be built upon by Hans, Johnoel, Jim I.
>>> and Ray Conser, who were assigned the lower priority topic of "code
>>> tuning"
>>> <http://admb-project.org/community/admb-meeting-march-29-31/ADMBMeetingReportv3.pdf#page=4>
>>> at the past ADMB meeting).
>>>
>>> I haven't really digested the results yet, because there are a lot
>>> of differences between the profiles I got for ADMB versions 9 and
>>> 10. However, here's a sample of the function calls that took up the
>>> most time, matched up with the times (in seconds) from version 9
>>> (compiled from the identical TPL file and identical inputs):
>>>
>>> v9.secs v10.secs function.name
>>> 2.62 14.8
>>> dvector::save_dvector_derivatives(dvar_vector_position const&) const
>>> 5.72 9.43 DF_FILE::fread(double const&)
>>> NA 9.1 dvector::operator()(int)
>>> 0.61 6.55 gradient_structure::jacobcalc(int,
>>> std::basic_ofstream<char, std::char_traits<char> > const&)
>>> 1.05 6.21
>>> restore_dvar_vector_derivatives(dvar_vector_position const&)
>>> 0.52 5.59 gradcalc(int, dvector const&)
>>> NA 5.08 dvector::elem(int)
>>> 1.66 4.33 DF_FILE::fwrite(double)
>>> 1.96 4.21 restore_dvar_vector_value(dvar_vector_position const&)
>>> NA 3.86 dvar_vector_position::indexmin() const
>>>
>>> These functions that are taking the most time seem to be deep in
>>> ADMB (or AUTODIF) and not stuff that's easy for a user to work
>>> around. If I'm not the only one finding version 10 to be slower than
>>> version 9, then I'm guessing that somebody else is better equipped
>>> to suggest what to do about it.
>>> -Ian
>>>
>>> On 11/12/2010 6:53 AM, dave fournier wrote:
>>>> Ian Taylor wrote:
>>>>
>>>> You could compile it for profiling and find out where the time is
>>>> spent
>>>> with gprof.
>>>>> After sending the yesterday, I caught the missing -O3. I tried
>>>>> adding it back and also just doing the compiling with the old
>>>>> script. But as long as I used the new libraries at the adlink
>>>>> stage, it went slow. And by slow, I mean more than 8 minutes for a
>>>>> model that previously took 2 1/2 minutes in ADMB version 9 if
>>>>> optimized and just over 3 minutes in safe mode (which also
>>>>> suggests that it's not just a matter of optimizing).
>>>>>
>>>>> I can try other models to see if the pattern I saw was consistent,
>>>>> but I'm also curious if anyone else has compared the speeds
>>>>> between versions. I don't know if this is in the documentation,
>>>>> but run times can be quantified with code like the following.
>>>>> -Ian
>>>>>
>>>>> TOP_OF_MAIN_SECTION
>>>>> time(&start);
>>>>>
>>>>> GLOBALS_SECTION
>>>>> #include <time.h>
>>>>> time_t start,finish;
>>>>> double elapsed_time;
>>>>>
>>>>> FINAL_SECTION
>>>>> time(&finish);
>>>>> elapsed_time = difftime(finish,start);
>>>>>
>>>>> On 11/12/2010 2:09 PM, Johnoel Ancheta wrote:
>>>>>> Opps... Sorry...
>>>>>>
>>>>>> I accidentally committed the Makefile without the -O3 option.
>>>>>>
>>>>>> I'll correct and commit the changes.
>>>>>>
>>>>>> Thanks Dave for catching that.
>>>>>>
>>>>>> Johnoel
>>>>>>
>>>>>> On 11/12/10 4:09 AM, dave fournier wrote:
>>>>>>> Ian Taylor wrote:
>>>>>>>
>>>>>>> Just a thought Re speed. I don't see a -O3 flag for gcc for
>>>>>>> optimization.
>>>>>>>> Hi Johnoel and others,
>>>>>>>>
>>>>>>>> Compiling ADMB 10 Beta on with 64-bit linux worked perfectly
>>>>>>>> for me with the files in the download.
>>>>>>>>
>>>>>>>> However, the new script you sent by email failed. From the
>>>>>>>> commands below (fuller notes attached), it looks like for
>>>>>>>> Ubuntu at least, it would be better to test for 64 vs 32 using
>>>>>>>> uname -m rather than uname -p.
>>>>>>>>
>>>>>>>> ~/admb/admb-10.0Beta $ uname -a
>>>>>>>> Linux ian-laptop 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16
>>>>>>>> 19:52:42 UTC 2010 x86_64 GNU/Linux
>>>>>>>> ~/admb/admb-10.0Beta $ uname -s
>>>>>>>> Linux
>>>>>>>> ~/admb/admb-10.0Beta $ uname -p
>>>>>>>> unknown
>>>>>>>> ~/admb/admb-10.0Beta $ uname -m
>>>>>>>> x86_64
>>>>>>>>
>>>>>>>> Aside from the good news of the successful install, this
>>>>>>>> version 10 Beta also produced identical results, which is also
>>>>>>>> good news. However, it seems to be running slower than version
>>>>>>>> 9, by a large margin, at least for a single model that I tried.
>>>>>>>> Perhaps related to this, when I compiled using GCC in and ADMB
>>>>>>>> version 9, the safe executable was about double the size of the
>>>>>>>> optimized. Compiling with ADMB version 10, the file sizes are
>>>>>>>> similar, and in between the old sizes. Perhaps the speed and
>>>>>>>> file size differences are related to changes in the compile
>>>>>>>> scripts? My old script was dated "23 May 2009".
>>>>>>>>
>>>>>>>> Finally, thank you, Johnoel, and all the others who've
>>>>>>>> contributed to the developments that have gone into this new
>>>>>>>> beta release.
>>>>>>>> -Ian
>>>>>>>>
>>>>>>>> On 11/10/2010 02:44 PM, Johnoel Ancheta wrote:
>>>>>>>>> Updated configure script is available.
>>>>>>>>>
>>>>>>>>> http://admb-project.googlecode.com/files/configure
>>>>>>>>>
>>>>>>>>> Download and replace the configure script in the admb-10.0Beta
>>>>>>>>> distribution.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 11/10/10 11:09 AM, Ben Bolker wrote:
>>>>>>>>>> (running under VMWare Fusion on Mac OS X.6, but I don't
>>>>>>>>>> think that
>>>>>>>>>> matters)
>>>>>>>>>>
>>>>>>>>>> It looks like configure is getting confused and thinking
>>>>>>>>>> my system is
>>>>>>>>>> 64-bit ... ?
>>>>>>>>>>
>>>>>>>>>>> uname -a
>>>>>>>>>> Linux ubuntu-10 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16
>>>>>>>>>> 19:48:22 UTC
>>>>>>>>>> 2010 i686 GNU/Linux
>>>>>>>>>>
>>>>>>>>>> uname -m
>>>>>>>>>> i686 ## indicates 32-bit kernel
>>>>>>>>>>
>>>>>>>>>> $ gcc -v
>>>>>>>>>> Using built-in specs.
>>>>>>>>>> Target: i486-linux-gnu
>>>>>>>>>> Configured with: ../src/configure -v --with-pkgversion='Ubuntu
>>>>>>>>>> 4.4.3-4ubuntu5'
>>>>>>>>>> --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
>>>>>>>>>> --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr
>>>>>>>>>> --enable-shared --enable-multiarch --enable-linker-build-id
>>>>>>>>>> --with-system-zlib --libexecdir=/usr/lib
>>>>>>>>>> --without-included-gettext
>>>>>>>>>> --enable-threads=posix
>>>>>>>>>> --with-gxx-include-dir=/usr/include/c++/4.4
>>>>>>>>>> --program-suffix=-4.4 --enable-nls --enable-clocale=gnu
>>>>>>>>>> --enable-libstdcxx-debug --enable-plugin --enable-objc-gc
>>>>>>>>>> --enable-targets=all --disable-werror --with-arch-32=i486
>>>>>>>>>> --with-tune=generic --enable-checking=release
>>>>>>>>>> --build=i486-linux-gnu
>>>>>>>>>> --host=i486-linux-gnu --target=i486-linux-gnu
>>>>>>>>>> Thread model: posix
>>>>>>>>>> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> sudo
>>>>>>>>>> ./configure
>>>>>>>>>> checking for gcc... gcc
>>>>>>>>>> checking whether the C compiler works... yes
>>>>>>>>>> checking for C compiler default output file name... a.out
>>>>>>>>>> checking for suffix of executables...
>>>>>>>>>> checking whether we are cross compiling... no
>>>>>>>>>> checking for suffix of object files... o
>>>>>>>>>> checking whether we are using the GNU C compiler... yes
>>>>>>>>>> checking whether gcc accepts -g... yes
>>>>>>>>>> checking for gcc option to accept ISO C89... none needed
>>>>>>>>>> checking for g++... g++
>>>>>>>>>> checking whether we are using the GNU C++ compiler... yes
>>>>>>>>>> checking whether g++ accepts -g... yes
>>>>>>>>>> checking how to run the C++ preprocessor... g++ -E
>>>>>>>>>> checking for grep that handles long lines and -e... /bin/grep
>>>>>>>>>> checking for egrep... /bin/grep -E
>>>>>>>>>> checking for ANSI C header files... yes
>>>>>>>>>> checking for sys/types.h... yes
>>>>>>>>>> checking for sys/stat.h... yes
>>>>>>>>>> checking for stdlib.h... yes
>>>>>>>>>> checking for string.h... yes
>>>>>>>>>> checking for memory.h... yes
>>>>>>>>>> checking for strings.h... yes
>>>>>>>>>> checking for inttypes.h... yes
>>>>>>>>>> checking for stdint.h... yes
>>>>>>>>>> checking for unistd.h... yes
>>>>>>>>>> checking iostream usability... yes
>>>>>>>>>> checking iostream presence... yes
>>>>>>>>>> checking for iostream... yes
>>>>>>>>>> checking for flex... flex
>>>>>>>>>> checking lex output file root... lex.yy
>>>>>>>>>> checking lex library... none needed
>>>>>>>>>> checking whether yytext is a pointer... no
>>>>>>>>>> checking for sed... yes
>>>>>>>>>> checking whether C++ compiler is 64-bit enabled... yes
>>>>>>>>>> checking for svnversion... yes
>>>>>>>>>> configure: creating ./config.status
>>>>>>>>>> config.status: creating GNUmakefile
>>>>>>>>>> config.status: creating admb_configure.h
>>>>>>>>>> config.status: creating scripts/bash/admb-bin
>>>>>>>>>>
>>>>>>>>>> Configure completed for admb10.0Beta-linux-gcc4.4.3-64bit
>>>>>>>>>> To build ADMB, type 'make'
>>>>>>>>>> bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$> sudo
>>>>>>>>>> make
>>>>>>>>>> CXX=g++ CXXFLAGS="-m64"
>>>>>>>>>> LIBPATH=../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects
>>>>>>>>>> DISK=../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist
>>>>>>>>>> ADMB_CONFIGURE=1
>>>>>>>>>> make --directory=src --file=linux.mak
>>>>>>>>>> make[1]: Entering directory `/usr/local/src/admb-10.0Beta/src'
>>>>>>>>>> rm -rf ../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist/{bin,lib,include,docs,docs/manuals,examples}
>>>>>>>>>>
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/linad99-olp
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/linad99-slp
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/nh99-olp
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/nh99-olp-stub
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/tools99-olp
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-slp
>>>>>>>>>>
>>>>>>>>>> mkdir -p
>>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp
>>>>>>>>>>
>>>>>>>>>> make --directory=df1b2-separable CC=gcc CXXFLAGS="-c -m64 -Wall
>>>>>>>>>> -Wno-deprecated -DUSE_LAPLACE -fpermissive -I../df1b2-separable
>>>>>>>>>> -I../nh99 -I../linad99 -I../tools99 -D__SPDLL__ -D__GNUDOS__
>>>>>>>>>> -Dlinux
>>>>>>>>>> -DOPT_LIB"
>>>>>>>>>> LIBPATH=../../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp
>>>>>>>>>>
>>>>>>>>>> DISKDIR=../../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist -f
>>>>>>>>>> optg32-rh8-laplace.mak disk
>>>>>>>>>> make[2]: Entering directory
>>>>>>>>>> `/usr/local/src/admb-10.0Beta/src/df1b2-separable'
>>>>>>>>>> g++ -c -m64 -Wall -Wno-deprecated -DUSE_LAPLACE -fpermissive
>>>>>>>>>> -I../df1b2-separable -I../nh99 -I../linad99 -I../tools99
>>>>>>>>>> -D__SPDLL__
>>>>>>>>>> -D__GNUDOS__ -Dlinux -DOPT_LIB df1b2impspf.cpp -o
>>>>>>>>>> ../../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp/df1b2impspf.obj
>>>>>>>>>>
>>>>>>>>>> In file included from /usr/include/features.h:378,
>>>>>>>>>> from /usr/include/unistd.h:26,
>>>>>>>>>> from ../linad99/fvar.hpp:173,
>>>>>>>>>> from ../nh99/admodel.h:69,
>>>>>>>>>> from df1b2impspf.cpp:8:
>>>>>>>>>> /usr/include/gnu/stubs.h:9:27: error: gnu/stubs-64.h: No such
>>>>>>>>>> file or
>>>>>>>>>> directory
>>>>>>>>>> df1b2impspf.cpp: In function ‘double
>>>>>>>>>> calculate_importance_sample_shess(const dvector&, const
>>>>>>>>>> dvector&, const
>>>>>>>>>> dmatrix&, const dvector&, const dvector&, const dmatrix&,
>>>>>>>>>> function_minimizer*)’:
>>>>>>>>>> df1b2impspf.cpp:20: warning: unused variable ‘Hessadjoint’
>>>>>>>>>> df1b2impspf.cpp:44: warning: unused variable ‘onvar’
>>>>>>>>>> df1b2impspf.cpp:49: warning: unused variable ‘j’
>>>>>>>>>> make[2]: *** [df1b2impspf.obj] Error 1
>>>>>>>>>> make[2]: Leaving directory
>>>>>>>>>> `/usr/local/src/admb-10.0Beta/src/df1b2-separable'
>>>>>>>>>> make[1]: *** [dist] Error 2
>>>>>>>>>> make[1]: Leaving directory `/usr/local/src/admb-10.0Beta/src'
>>>>>>>>>> make: *** [all] Error 2
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Developers mailing list
>>>>>>>>>> Developers at admb-project.org
>>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Developers mailing list
>>>>>>>>> Developers at admb-project.org
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Developers mailing list
>>>>>>>> Developers at admb-project.org
>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>
>>>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Developers mailing list
>>> Developers at admb-project.org
>>> http://lists.admb-project.org/mailman/listinfo/developers
>>
>
> _______________________________________________
> Developers mailing list
> Developers at admb-project.org
> http://lists.admb-project.org/mailman/listinfo/developers
>
--
John Sibert
Emeritus Researcher, SOEST
University of Hawaii at Manoa
Visit the ADMB project http://admb-project.org/
More information about the Developers
mailing list