[Developers] Fwd: [New comment] Automatic Differentiation: The most criminally underused tool in the potential machine learning toolbox?

[John Sibert]

-------- Original Message --------
Subject: 	[New comment] Automatic Differentiation: The most criminally 
underused tool in the potential machine learning toolbox?
Date: 	Fri, 30 Nov 2012 04:56:22 +0000
From: 	Justin Domkes Weblog <comment-reply at wordpress.com>
Reply-To: 	Justin Domkes Weblog 
<comment+_dvt62oan5kqp1btsg5w-sp301gstz6ua-9w7kz9 at comment.wordpress.com>
To: 	sibert at hawaii.edu



WordPress.com
Respond to this comment by replying above this line


    New comment on *Justin Domke's Weblog
    <http://justindomke.wordpress.com>*

	<http://justindomke.wordpress.com>

<http://gravatar.com/lingpipe> 	

*Bob Carpenter* commented 
<http://justindomke.wordpress.com/2009/02/17/automatic-differentiation-the-most-criminally-underused-tool-in-the-potential-machine-learning-toolbox#comment-1729> 
on Automatic Differentiation: The most criminally underused tool in the 
potential machine learning toolbox? 
<http://justindomke.wordpress.com/2009/02/17/automatic-differentiation-the-most-criminally-underused-tool-in-the-potential-machine-learning-toolbox>. 


in response to *justindomke*:

    I recently got back reviews of a paper in which I used automatic
    differentiation. Therein, a reviewer clearly thought I was using
    finite difference, or “numerical” differentiation. This has led me
    to wondering: Why don’t machine learning people use automatic
    differentiation more?  Why don’t they use it…constantly? Before
    recklessly speculating on the answer, let me [...]

Justin — I’m following up to let you know we finally released Stan, our 
adaptive Hamiltonian Monte Carlo sampler:

http://mc-stan.org/

It has a BSD license and we’re hoping people feel free to use all or 
parts of it in their own work.

It’s written in C++ with taped reverse-mode auto-dif implemented with 
templates, and fully integrated with all the C++ special functions and 
with Eigen and with our own probability function library. Our auto-dif’s 
both more extensible and faster than CppAD or Sacado. I still need to 
write doc on the auto-dif itself and then put a link up on autodiff.org 
and on the Wikipedia page; but if you know auto-dif, it’s very 
straightforward, and our unit tests provide extensive simple examples of 
its use.

It also has a programming language like WinBUGS that lets users express 
models more naturally, including expressing constraints for which 
Jacobian adjustments are automatically calculated (and auto-diffed).

Oh, and there’s a complete R interface (Python, MATLAB and Julia are on 
the to-do list, too). There’s a 250 page manual with many statistical 
modeling examples and a full definition of the language and its functions.

The next release (out in a week or two) will add comparison operators, 
logical operators, and Newton-Raphson optimization. We’ll probably put 
in the quasi-Newton L-BFGS sooner or later. We’re also planning to do 
forward-mode templated auto-dif in the usual way so that we can 
calculate Hessians for both optimization and Riemann Manifold HMC. We 
got an NSF grant to continue work on it, so we’ll be able to continue to 
add to its functionality and support it.

P.S. Even for pretty hairy functions, our compile times are a lot better 
than you reported in the follow-on post. But they’re still pretty bad. 
clang++ is much better than g++ in terms of compile times.

Reply 
<http://justindomke.wordpress.com/2009/02/17/automatic-differentiation-the-most-criminally-underused-tool-in-the-potential-machine-learning-toolbox?replytocom=1729#respond> 
Comments 
<http://justindomke.wordpress.com/2009/02/17/automatic-differentiation-the-most-criminally-underused-tool-in-the-potential-machine-learning-toolbox> 


*Want less email?* Modify your Subscription Options 
<https://subscribe.wordpress.com/?key=4220e3bc2f2ce5bd9d8f568e895ce9d4&email=sibert%40hawaii.edu&option=comments>. 



Thanks for flying with WordPress.com <http://wordpress.com>

Bob Carpenter commented: "Justin -- I'm following up to let you know we 
finally released Stan, our adaptive Hamiltonian Monte Carlo sampler: 
http://mc-stan.org/ It has a BSD license and we're hoping people feel 
free to use all or parts of it in their own work. It's writte"