[Developers] profiling ADMB model with gprof
Ian Taylor
Ian.Taylor at noaa.gov
Tue Nov 16 14:31:07 PST 2010
On 11/15/2010 10:52 AM, dave fournier wrote:
> I suspect the difference could be in writing to
> temporary files.
I didn't see any evidence of temporary files
getting written. I did notice that the compile and
link commands for v10beta didn't have a -m64
command, but adding it in didn't seem to make any
difference. Indeed, the only thing that seems to
make a difference on speed was whether I was
linking to the new libraries or the old.
Here's some output from an additional test that I
just conducted with the "catage" example included
with ADMB, where I modified the TPL to provide
iternations and run times and I modified the data
file to include 200 years and 40 ages so that it
would run slower (files are attached). The
commands below are identical with the exception of
whether they point to the v9 or v10 directories.
As you can see from the last section, the
temporary files were never used and the run times
were 17 seconds in v9 and 50 seconds in v10. (This
was all done with GCC 4.1.2 on 64 bit Red Hat
Enterprise Linux Server release 5.5 with 3 GHz
processor and 32 GB memory.) The results continue
to be identical, which is an important first step.
However, I think I'll stick with the old code
until I hear news that somebody else has achieved
similar performance between versions 9 and 10.
-Ian
[catage2]$ # convert
[catage2]$ tpl2cpp catage2_v9
[catage2]$ tpl2cpp catage2_v10
[catage2]$
[catage2]$ # compile
[catage2]$ g++ -c -m64 -O3 -Wno-deprecated
-D__GNUDOS__ -Dlinux -DOPT_LIB -DUSE_LAPLACE
-fpermissive -I.
-I/home/itaylor/h_itaylor/admb/v9/include
catage2_v9.cpp
[catage2]$ g++ -c -m64 -O3 -Wno-deprecated
-D__GNUDOS__ -Dlinux -DOPT_LIB -DUSE_LAPLACE
-fpermissive -I.
-I/home/itaylor/h_itaylor/admb/v10/include
catage2_v10.cpp
[catage2]$
[catage2]$ # link
[catage2]$ g++ -m64
-L/home/itaylor/h_itaylor/admb/v9/lib
catage2_v9.o -ldf1b2o -ladmod -ladt -lado
-ldf1b2o -ladmod -ladt -lado -o catage2_v9
[catage2]$ g++ -m64
-L/home/itaylor/h_itaylor/admb/v10/lib
catage2_v10.o -ldf1b2o -ladmod -ladt -lado
-ldf1b2o -ladmod -ladt -lado -o catage2_v10
[catage2]$
[catage2]$ # run
[catage2]$ ./catage2_v9 -gbs 1000000000 -cbs
1000000000 -nox > run_notes_v9.txt
[catage2]$ ./catage2_v10 -gbs 1000000000 -cbs
1000000000 -nox > run_notes_v10.txt
[catage2]$
[catage2]$ # check for temporary file useage
[catage2]$ cat catage2_v9.log
size of file gradfil1.tmp = 0
size of file gradfil2.tmp = 0
size of file varssave.tmp = 0
size of file cmpdiff.tmp = 0
[catage2]$ cat catage2_v10.log
size of file gradfil1.tmp = 0
size of file gradfil2.tmp = 0
size of file varssave.tmp = 0
size of file cmpdiff.tmp = 0
[catage2]$
[catage2]$ # run times
[catage2]$ head -2 catage2_v9.rep
1351 # N iternations
17 # runtime (sec)
[catage2]$ head -2 catage2_v10.rep
1351 # N iternations
50 # runtime (sec)
>
>> The suggestion of trying gprof is something
>> that I'd been meaning to do in the past, and
>> now there was an incentive.
>> I changed the subject line of this email to
>> better reflect that this is now the topic.
>>
>> I haven't seen any instructions on using gprof
>> for ADMB, so I posted the steps that eventually
>> succeeded at
>> http://admb-project.org/community/tutorials-and-examples/how-to-use-gprof-the-gnu-profiler.
>> If it turns out that these options were already
>> built into the existing scripts or if there's
>> an easier way, let me know so I can update the
>> instructions on the web. (If the instructions
>> are any good, then maybe they could be built
>> upon by Hans, Johnoel, Jim I. and Ray Conser,
>> who were assigned the lower priority topic of
>> "code tuning"
>> <http://admb-project.org/community/admb-meeting-march-29-31/ADMBMeetingReportv3.pdf#page=4>
>> at the past ADMB meeting).
>>
>> I haven't really digested the results yet,
>> because there are a lot of differences between
>> the profiles I got for ADMB versions 9 and 10.
>> However, here's a sample of the function calls
>> that took up the most time, matched up with the
>> times (in seconds) from version 9 (compiled
>> from the identical TPL file and identical inputs):
>>
>> v9.secs v10.secs function.name
>> 2.62 14.8
>> dvector::save_dvector_derivatives(dvar_vector_position
>> const&) const
>> 5.72 9.43 DF_FILE::fread(double const&)
>> NA 9.1 dvector::operator()(int)
>> 0.61 6.55
>> gradient_structure::jacobcalc(int,
>> std::basic_ofstream<char,
>> std::char_traits<char> > const&)
>> 1.05 6.21
>> restore_dvar_vector_derivatives(dvar_vector_position
>> const&)
>> 0.52 5.59 gradcalc(int, dvector const&)
>> NA 5.08 dvector::elem(int)
>> 1.66 4.33 DF_FILE::fwrite(double)
>> 1.96 4.21
>> restore_dvar_vector_value(dvar_vector_position
>> const&)
>> NA 3.86
>> dvar_vector_position::indexmin() const
>>
>> These functions that are taking the most time
>> seem to be deep in ADMB (or AUTODIF) and not
>> stuff that's easy for a user to work around. If
>> I'm not the only one finding version 10 to be
>> slower than version 9, then I'm guessing that
>> somebody else is better equipped to suggest
>> what to do about it.
>> -Ian
>>
>> On 11/12/2010 6:53 AM, dave fournier wrote:
>>> Ian Taylor wrote:
>>>
>>> You could compile it for profiling and find
>>> out where the time is spent
>>> with gprof.
>>>> After sending the yesterday, I caught the
>>>> missing -O3. I tried adding it back and also
>>>> just doing the compiling with the old script.
>>>> But as long as I used the new libraries at
>>>> the adlink stage, it went slow. And by slow,
>>>> I mean more than 8 minutes for a model that
>>>> previously took 2 1/2 minutes in ADMB version
>>>> 9 if optimized and just over 3 minutes in
>>>> safe mode (which also suggests that it's not
>>>> just a matter of optimizing).
>>>>
>>>> I can try other models to see if the pattern
>>>> I saw was consistent, but I'm also curious if
>>>> anyone else has compared the speeds between
>>>> versions. I don't know if this is in the
>>>> documentation, but run times can be
>>>> quantified with code like the following.
>>>> -Ian
>>>>
>>>> TOP_OF_MAIN_SECTION
>>>> time(&start);
>>>>
>>>> GLOBALS_SECTION
>>>> #include <time.h>
>>>> time_t start,finish;
>>>> double elapsed_time;
>>>>
>>>> FINAL_SECTION
>>>> time(&finish);
>>>> elapsed_time = difftime(finish,start);
>>>>
>>>> On 11/12/2010 2:09 PM, Johnoel Ancheta wrote:
>>>>> Opps... Sorry...
>>>>>
>>>>> I accidentally committed the Makefile
>>>>> without the -O3 option.
>>>>>
>>>>> I'll correct and commit the changes.
>>>>>
>>>>> Thanks Dave for catching that.
>>>>>
>>>>> Johnoel
>>>>>
>>>>> On 11/12/10 4:09 AM, dave fournier wrote:
>>>>>> Ian Taylor wrote:
>>>>>>
>>>>>> Just a thought Re speed. I don't see a -O3
>>>>>> flag for gcc for optimization.
>>>>>>> Hi Johnoel and others,
>>>>>>>
>>>>>>> Compiling ADMB 10 Beta on with 64-bit
>>>>>>> linux worked perfectly for me with the
>>>>>>> files in the download.
>>>>>>>
>>>>>>> However, the new script you sent by email
>>>>>>> failed. From the commands below (fuller
>>>>>>> notes attached), it looks like for Ubuntu
>>>>>>> at least, it would be better to test for
>>>>>>> 64 vs 32 using uname -m rather than uname -p.
>>>>>>>
>>>>>>> ~/admb/admb-10.0Beta $ uname -a
>>>>>>> Linux ian-laptop 2.6.32-25-generic
>>>>>>> #45-Ubuntu SMP Sat Oct 16
>>>>>>> 19:52:42 UTC 2010 x86_64 GNU/Linux
>>>>>>> ~/admb/admb-10.0Beta $ uname -s
>>>>>>> Linux
>>>>>>> ~/admb/admb-10.0Beta $ uname -p
>>>>>>> unknown
>>>>>>> ~/admb/admb-10.0Beta $ uname -m
>>>>>>> x86_64
>>>>>>>
>>>>>>> Aside from the good news of the successful
>>>>>>> install, this version 10 Beta also
>>>>>>> produced identical results, which is also
>>>>>>> good news. However, it seems to be running
>>>>>>> slower than version 9, by a large margin,
>>>>>>> at least for a single model that I tried.
>>>>>>> Perhaps related to this, when I compiled
>>>>>>> using GCC in and ADMB version 9, the safe
>>>>>>> executable was about double the size of
>>>>>>> the optimized. Compiling with ADMB version
>>>>>>> 10, the file sizes are similar, and in
>>>>>>> between the old sizes. Perhaps the speed
>>>>>>> and file size differences are related to
>>>>>>> changes in the compile scripts? My old
>>>>>>> script was dated "23 May 2009".
>>>>>>>
>>>>>>> Finally, thank you, Johnoel, and all the
>>>>>>> others who've contributed to the
>>>>>>> developments that have gone into this new
>>>>>>> beta release.
>>>>>>> -Ian
>>>>>>>
>>>>>>> On 11/10/2010 02:44 PM, Johnoel Ancheta
>>>>>>> wrote:
>>>>>>>> Updated configure script is available.
>>>>>>>>
>>>>>>>> http://admb-project.googlecode.com/files/configure
>>>>>>>>
>>>>>>>>
>>>>>>>> Download and replace the configure script
>>>>>>>> in the admb-10.0Beta distribution.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/10/10 11:09 AM, Ben Bolker wrote:
>>>>>>>>> (running under VMWare Fusion on Mac
>>>>>>>>> OS X.6, but I don't think that
>>>>>>>>> matters)
>>>>>>>>>
>>>>>>>>> It looks like configure is getting
>>>>>>>>> confused and thinking my system is
>>>>>>>>> 64-bit ... ?
>>>>>>>>>
>>>>>>>>>> uname -a
>>>>>>>>> Linux ubuntu-10 2.6.32-25-generic
>>>>>>>>> #45-Ubuntu SMP Sat Oct 16 19:48:22 UTC
>>>>>>>>> 2010 i686 GNU/Linux
>>>>>>>>>
>>>>>>>>> uname -m
>>>>>>>>> i686 ## indicates 32-bit kernel
>>>>>>>>>
>>>>>>>>> $ gcc -v
>>>>>>>>> Using built-in specs.
>>>>>>>>> Target: i486-linux-gnu
>>>>>>>>> Configured with: ../src/configure -v
>>>>>>>>> --with-pkgversion='Ubuntu
>>>>>>>>> 4.4.3-4ubuntu5'
>>>>>>>>> --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs
>>>>>>>>>
>>>>>>>>> --enable-languages=c,c++,fortran,objc,obj-c++
>>>>>>>>> --prefix=/usr
>>>>>>>>> --enable-shared --enable-multiarch
>>>>>>>>> --enable-linker-build-id
>>>>>>>>> --with-system-zlib --libexecdir=/usr/lib
>>>>>>>>> --without-included-gettext
>>>>>>>>> --enable-threads=posix
>>>>>>>>> --with-gxx-include-dir=/usr/include/c++/4.4
>>>>>>>>> --program-suffix=-4.4 --enable-nls
>>>>>>>>> --enable-clocale=gnu
>>>>>>>>> --enable-libstdcxx-debug --enable-plugin
>>>>>>>>> --enable-objc-gc
>>>>>>>>> --enable-targets=all --disable-werror
>>>>>>>>> --with-arch-32=i486
>>>>>>>>> --with-tune=generic
>>>>>>>>> --enable-checking=release
>>>>>>>>> --build=i486-linux-gnu
>>>>>>>>> --host=i486-linux-gnu
>>>>>>>>> --target=i486-linux-gnu
>>>>>>>>> Thread model: posix
>>>>>>>>> gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$
>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>> sudo ./configure
>>>>>>>>> checking for gcc... gcc
>>>>>>>>> checking whether the C compiler works...
>>>>>>>>> yes
>>>>>>>>> checking for C compiler default output
>>>>>>>>> file name... a.out
>>>>>>>>> checking for suffix of executables...
>>>>>>>>> checking whether we are cross
>>>>>>>>> compiling... no
>>>>>>>>> checking for suffix of object files... o
>>>>>>>>> checking whether we are using the GNU C
>>>>>>>>> compiler... yes
>>>>>>>>> checking whether gcc accepts -g... yes
>>>>>>>>> checking for gcc option to accept ISO
>>>>>>>>> C89... none needed
>>>>>>>>> checking for g++... g++
>>>>>>>>> checking whether we are using the GNU
>>>>>>>>> C++ compiler... yes
>>>>>>>>> checking whether g++ accepts -g... yes
>>>>>>>>> checking how to run the C++
>>>>>>>>> preprocessor... g++ -E
>>>>>>>>> checking for grep that handles long
>>>>>>>>> lines and -e... /bin/grep
>>>>>>>>> checking for egrep... /bin/grep -E
>>>>>>>>> checking for ANSI C header files... yes
>>>>>>>>> checking for sys/types.h... yes
>>>>>>>>> checking for sys/stat.h... yes
>>>>>>>>> checking for stdlib.h... yes
>>>>>>>>> checking for string.h... yes
>>>>>>>>> checking for memory.h... yes
>>>>>>>>> checking for strings.h... yes
>>>>>>>>> checking for inttypes.h... yes
>>>>>>>>> checking for stdint.h... yes
>>>>>>>>> checking for unistd.h... yes
>>>>>>>>> checking iostream usability... yes
>>>>>>>>> checking iostream presence... yes
>>>>>>>>> checking for iostream... yes
>>>>>>>>> checking for flex... flex
>>>>>>>>> checking lex output file root... lex.yy
>>>>>>>>> checking lex library... none needed
>>>>>>>>> checking whether yytext is a pointer... no
>>>>>>>>> checking for sed... yes
>>>>>>>>> checking whether C++ compiler is 64-bit
>>>>>>>>> enabled... yes
>>>>>>>>> checking for svnversion... yes
>>>>>>>>> configure: creating ./config.status
>>>>>>>>> config.status: creating GNUmakefile
>>>>>>>>> config.status: creating admb_configure.h
>>>>>>>>> config.status: creating
>>>>>>>>> scripts/bash/admb-bin
>>>>>>>>>
>>>>>>>>> Configure completed for
>>>>>>>>> admb10.0Beta-linux-gcc4.4.3-64bit
>>>>>>>>> To build ADMB, type 'make'
>>>>>>>>> bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$
>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>> <mailto:bolker at ubuntu-10:/usr/local/src/admb-10.0Beta$>
>>>>>>>>> sudo make
>>>>>>>>> CXX=g++ CXXFLAGS="-m64"
>>>>>>>>> LIBPATH=../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects
>>>>>>>>>
>>>>>>>>> DISK=../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist
>>>>>>>>> ADMB_CONFIGURE=1
>>>>>>>>> make --directory=src --file=linux.mak
>>>>>>>>> make[1]: Entering directory
>>>>>>>>> `/usr/local/src/admb-10.0Beta/src'
>>>>>>>>> rm -rf
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist/{bin,lib,include,docs,docs/manuals,examples}
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/linad99-olp
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/linad99-slp
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/nh99-olp
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/nh99-olp-stub
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/tools99-olp
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-slp
>>>>>>>>>
>>>>>>>>> mkdir -p
>>>>>>>>> ../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp
>>>>>>>>>
>>>>>>>>> make --directory=df1b2-separable CC=gcc
>>>>>>>>> CXXFLAGS="-c -m64 -Wall
>>>>>>>>> -Wno-deprecated -DUSE_LAPLACE
>>>>>>>>> -fpermissive -I../df1b2-separable
>>>>>>>>> -I../nh99 -I../linad99 -I../tools99
>>>>>>>>> -D__SPDLL__ -D__GNUDOS__ -Dlinux
>>>>>>>>> -DOPT_LIB"
>>>>>>>>> LIBPATH=../../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp
>>>>>>>>>
>>>>>>>>> DISKDIR=../../build/admb10.0Beta-linux-gcc4.4.3-64bit/dist
>>>>>>>>> -f
>>>>>>>>> optg32-rh8-laplace.mak disk
>>>>>>>>> make[2]: Entering directory
>>>>>>>>> `/usr/local/src/admb-10.0Beta/src/df1b2-separable'
>>>>>>>>>
>>>>>>>>> g++ -c -m64 -Wall -Wno-deprecated
>>>>>>>>> -DUSE_LAPLACE -fpermissive
>>>>>>>>> -I../df1b2-separable -I../nh99
>>>>>>>>> -I../linad99 -I../tools99 -D__SPDLL__
>>>>>>>>> -D__GNUDOS__ -Dlinux -DOPT_LIB
>>>>>>>>> df1b2impspf.cpp -o
>>>>>>>>> ../../build/admb10.0Beta-linux-gcc4.4.3-64bit/objects/df1b2-separable-olp/df1b2impspf.obj
>>>>>>>>>
>>>>>>>>> In file included from
>>>>>>>>> /usr/include/features.h:378,
>>>>>>>>> from
>>>>>>>>> /usr/include/unistd.h:26,
>>>>>>>>> from
>>>>>>>>> ../linad99/fvar.hpp:173,
>>>>>>>>> from
>>>>>>>>> ../nh99/admodel.h:69,
>>>>>>>>> from df1b2impspf.cpp:8:
>>>>>>>>> /usr/include/gnu/stubs.h:9:27: error:
>>>>>>>>> gnu/stubs-64.h: No such file or
>>>>>>>>> directory
>>>>>>>>> df1b2impspf.cpp: In function ‘double
>>>>>>>>> calculate_importance_sample_shess(const
>>>>>>>>> dvector&, const dvector&, const
>>>>>>>>> dmatrix&, const dvector&, const
>>>>>>>>> dvector&, const dmatrix&,
>>>>>>>>> function_minimizer*)’:
>>>>>>>>> df1b2impspf.cpp:20: warning: unused
>>>>>>>>> variable ‘Hessadjoint’
>>>>>>>>> df1b2impspf.cpp:44: warning: unused
>>>>>>>>> variable ‘onvar’
>>>>>>>>> df1b2impspf.cpp:49: warning: unused
>>>>>>>>> variable ‘j’
>>>>>>>>> make[2]: *** [df1b2impspf.obj] Error 1
>>>>>>>>> make[2]: Leaving directory
>>>>>>>>> `/usr/local/src/admb-10.0Beta/src/df1b2-separable'
>>>>>>>>>
>>>>>>>>> make[1]: *** [dist] Error 2
>>>>>>>>> make[1]: Leaving directory
>>>>>>>>> `/usr/local/src/admb-10.0Beta/src'
>>>>>>>>> make: *** [all] Error 2
>>>>>>>>> _______________________________________________
>>>>>>>>>
>>>>>>>>> Developers mailing list
>>>>>>>>> Developers at admb-project.org
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>> Developers mailing list
>>>>>>>> Developers at admb-project.org
>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> Developers mailing list
>>>>>>> Developers at admb-project.org
>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>> <mailto:Developers at admb-project.org>
>>>>>>> http://lists.admb-project.org/mailman/listinfo/developers
>>>>>>>
>>>>>>
>>>>>
>>>
>> ------------------------------------------------------------------------
>>
>>
>> _______________________________________________
>> Developers mailing list
>> Developers at admb-project.org
>> http://lists.admb-project.org/mailman/listinfo/developers
>>
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: catage2.dat
URL: <http://lists.admb-project.org/pipermail/developers/attachments/20101116/e2c66cfb/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: catage2.tpl
URL: <http://lists.admb-project.org/pipermail/developers/attachments/20101116/e2c66cfb/attachment-0003.ksh>
More information about the Developers
mailing list