Numerical Issues in Statistical Computing for the Social Scientist --
Resources for Numerical Analysis and Numerical Algorithms

Last Updated: June 5, 2008

Books

• "Numerical Issues in Statistical Computing for the Social Scientist, Micah Altman, Jeff Gill, Michael McDonald. Also see our accompanying software
• Books by James Gentle: Numerical Linear Algebra for Applications in Statistics, Elements of Computational Statistics, Handbook of Computational Statistics, Random Number Generation and Monte Carlo Methods
• Accuracy and Stability of Numerical Algorithms, Nicholas J. Higham
• Numerical Optimization, Jorge Nocedal and Stephen J. Wright

Journals

• Journal of Statistical Software
• Computational Statistics and Data Analysis
• Computational Statistics
• Computational Statistics -- Proceedings of the xx-th Symposium on Computational Statistics (COMPSTAT)
• Transactions on Mathematical Software
• The Journal of Statistical Computation and Simulation
• The American Statistician (focussed on general statistics, but also publishes on statistical computing)
• Journal of Computational and Graphical Statistics
• Statistics and Computing
• Applied Mathematics and Computing
• Combinatorics, Probability and Computing
• Computational Economics
• ACM Transactions on Modeling and Computer Simulations (TOMACS)
• SIAM Journals (on Numerical Analysis; Optimization; Scientific Computing)
• Communications in Statistics - Simulation and Computation
• Computing Science and Statistics (proceedings of Interface XX)
• Journal of the Japanese Society of Computational Statistics

News and Mailing-Lists

• NA-Net: a netlib based numerical analysis e-mail list
• Chance News: Devoted to discussion of randomness
• OON List: Devoted to object oriented numeric processing
• Mathematics Usenet Newsgroups: sci.math,sci.math.num-analysis,sci.math.research,sci.math.symbolic, sci.nonlinear , sci.op-research
• Statistical Computing and Graphics (ASA Section Newsletter)

High Precision Numerical Software

• Software for Evaluating the Accuracy and Reliability of Research

  • Accuracy perturbation tests for evaluating the accuracy of statistical results, fingerprints for validating statistical files, and global optimality tests
  • PRECIS Tools to explore stability, and measure backward error of computations.
  • INTLAB is an interval arithmetic module for MATLAB. It can be used for measuring the error bounds of some Matlab computations.
  • MUMPS - A parallel sparce matrix solver with backward error analysis
    
  • Pseudospectra Tools for analytical and graphical investigation of ill-conditioned matrices.
  • UNF fingerprints for validating statistical files, and global optimality tests

• General High Precision Numerical Software and Add-On's

  • Mathematica and Other CAS's. The premier commercial package for high precision calculations is Mathematica, which provides extensive support for arbitrary precision arithmetic, optimization, and interval arithmetic. The commercial packages Maple and Matlab also have strong followings.
    
  • Open source matrix algebra and CAS's: There are many alternatives to Maple, Matlab, and Mathematica, -- some will do interval and arbitrary-precision arithmet. S ee our free software guide.
    
  • Multiple and Arbitrary Precision Arithmetic: bc is a simple to use arbitrary-precision calculator available on most linux systems. GMP, MPFR, APMath, and CLN,  APFP are free libraries for arbitrary precision arithmetic in C++, Java and Ruby.
    
  • Tools for Automatic Differentiation: tools specifically for automated differentiation, to be used where hand-coding analytic gradients is not practical.
    

• High Precision Statistical Distribution Libraries

  • ELV: Computation of Statistical Distributions (ELV). A DOS program for computing a variety of statistical distributions precisely.
  • DCDFLIB & DCDFSTAT: High precision statistical algorithms from the ACM and other sources implemented by Barry Brown, under a grant from the National Cancer Institute. (Also available from Netlib)
    

• Statistical Software and Plug-ins for statistical Software
  

  See "The Impoverished Social Scientist's Guide to Free Statistical Software and Resources" for an extensive list of non-commercial statistical software packages, and links to lists of commercial packages.

High Performance Software for Statistical Computing

• High Performance Libraries
  
  • q
    q
  • Atlas - Automatically tuned linear algebra system. Can be used almost anywere BLAS is used.
    
  • NumPy - python libraries for numerically intensive computing (Also see SciPy for a set of statistical and mathematical tools)
    
  • PDL - perl data library, for manipulating large matrices in PERL
  • GSL : The gnu Scientific Library
  • Java Numeric Code JAIDA is a set of libraries for plots, historgrams, curve fitting, and related analysis. JAMA is useful for matrix algebraColt is a broad set of librares for scientfic computing, including random number generation, statistics, and linear algebra.
  • LibOIL A C++ library of basic numerical codes, optimized in assembly for a variety of target cpu architechtures
• Parallelization Frameworks
  
  • Condor - a parallelization framework specializing in cycle-harvesting and embarrassingly parallel jobs. Also supports PVM and MPI interfaces (with some limits).
  • Globus - Heavy duty open source grid computing framework. (Also see Instant-Grid a Knoppix-based distribution that makes it easy to set up a grid quickly from bootable CD's)
  • Mosix - a Single System Image ("form and forget") parallelization framework. Automatically distributes processes across a cluster. Supports computing grids as well.
  • MPI - Message Passing Interface, a standard message-based parallelization framework
  • Networkspaces -- distributed shared memory model parallelization framework for R , Python, Matlab.
    
  • OpenMosix - An Open source implementation of MOSIX (with fewer features) (Also see ClusterKnoppix a Knoppix-based distribution that makes it easy to set up an Open Mosix cluster quickly from bootable CD's)
  • PVM - Parallel Virtual Machine, a standard message-based parallelization framework
  • Scalr An open source implementation of Amazon's E3 clustering framework supporting the same APIs.
  • Snow - A parallelization framework for R. Can be used on its own, or on top of MPI or PVM. (Also see SnowFT at CRAN for a fault tolerant version.)
  • UPC - Unified parallel C. An explict parallelization framework for local and distributed computing. Compatible with GCC.


• System Performance Tuning and Code Performance Analysis (Profiling) Tools
  • butler - a benchmarking tool for R
  • CHUD- profiling tools for Apple OS X, includes SHARK
  • codeutils - more detailed code profiling for R
  • Dstat - Integrated system resource usage statistics for Linux
  • Systat - Basic system resource usage statistics for Linux:
    
  • gprof - standard gcc profiler
  • hprof - Sun's java cpu & memory profiler
  • JIP-interactive java proiler
  • Jprof - Java profiler
  • Oprofile - OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low over
  • Performance Inspector - A suite of tools for Linux to profile kernal and application performance.
    
  • Perfsuite - performance analysis for thread and MPI based parallel programs
    
  • PPW parallel performance wizard, performance analysis for shared memory and UPC parallelization
  • Profmon - process performance statistics for Windows
  • Strace - system call traces and timings for Linux
  • TAU - TAU (Tuning and Analysis Utilities) is a portable profiling and tracing toolkit for performance analysis of parallel programs written in Fortran, C, C++, Java, Python.
  • Vtune - extensive code performance analysis for INTEL processors
  • Yourkit - a tuning and profiling application for java
    

Optimization Resources

• EMOO Evolutionary multi objective optimization
• Global Optimization Links: Optimization links, collected by Arnold Neumaier, Computational Mathematics, University of Vienna
• INFORMS OR/MS Resource Collection: This is a page for pointers to all aspects of operations research.
• NCSA: offers software and supercomputer time for projects
• Optimization Decision Tree, by Mittelmann and Spellucci
• Optimization FAQS: Links to references and software for linear and nonlinear programming.
• OTC: Optimization technology center. Includes a networked optimization testbed, where you can submit problems to an optimization server for solution.

Test Data and Test Matrices for Statistical Software

• STRD: Statistical reference datasets. A repository of reference data sets with certified computational results that enable the objective evaluation of statistical software.
• Matrix Market: A repository of test data for use in comparative studies of algorithms for numerical linear algebra. (A paper describing this resource is also available.)

Random Number Generators and Test Suites

• Randomness Resources
  • PLAB: A Server on the Theory and Practice of Random Number Generation. A server run by a team of mathematicians and computer scientists at the University of Salzburg's Mathematics Department. Contains tutorials, pre-prints, and software for generators and testing.
  • Random number resources for cryptography
• Test Suites
  • TestU01: The most complete testing suite for serial PRNG's.
  • Die Harder: A replacement and extension for the venerable diehard tools for PRNG testing
  • DIEHARD: A test suite for random number generators, by George Marsaglia. Once very popular, but now surpassed by testu01. This site also contains 600MB of high-quality random numbers, as downloadable files.
  • SPRNG: A Scalable Library for Pseudorandom Number Generation. Primarily for parallelizable random number generation -- and also contains some tests not found in any other package, and tests tailored to parallel generation.
• Random Number Generators
  • Hot-Bits: radioactive bits (also see the accuracy package for R which can obtain bits automatically from hot-bits.)
  • /dev/random is provided in Linux, and gathers entropy from hardware interrupts.
  • HAVEGE a sophisticated non-kernel level hardware entropy gathering system with a pseudo-random expansion
  • SPRNG: A Scalable Library for Pseudorandom Number Generation. Test suites for random number generators by the National Center for Supercomputing Applications (NCSA). Specially designed for generating and testing parallel random number generators.
  • Lava-lamps: retro-random
  • Random.org: random bits from the atmosphere
  • Commercial Hardware Random Number Generators: A number of companies offer PC (and other) hardware that they claim generates "true" random numbers through stochastic physical processes: COMSCIRE, Protego Information, Aware (radioactive-decay random generators).
  • Intel 8xx series random number generation. These popular Intel chipsets provide true random number generators in hardware that sample from thermal noise. (See this Intel Random Number Generator Whitepaper.) Unfortunately, most statistical packages do not provide direct access to this generator on hardware that supports it. To access this widely available hardware generator under MSWindows, use the Microsoft CryptoAPI as decribed in this Intel white paper. Under Linux, install the intel-rng-tools drivers from the gkernel project, and this source will automatically be added to the /dev/random entropy pool.

General Resources for Numerical and Scientific Algorithms

• Prof. Hossein Arsham algorithm tutorials and links
• CALGO: Collected Algorithms of the Association of Communication Machinery. Software associated with publications in Transactions on Mathematical Software and other ACM journals.
• CGAL Computational Geometric Algorithm Library
• GAMS: Guide to Available Mathematical Software. An extensive cross-index and virtual repository of mathematical and statistical software for use in computational science and engineering.
• INRIA : Tools for scientific computing
• JASA Archive: A (small) archive of software associated with publications in the Journal of the American Statistical Association
• LCIC : Linux Clustering Information Center
  
• MAS Scien software links
  
• Mathtools: Large collection of links to computing software and articles (registration required)
• Netlib: A repository for mathematical and simulation code, including Linpack and Lapack
• NHSE: National High Performance Software Exchange Repository. Documents, data, and information of interest to the high performance and parallel computing community.
• Numerical recipes: The Numerical Recipes books provides useful introductions to a variety of numeric algorithms. (Venerable, but not uncriticized (also here)
• Numerical Analysis FAQ: A summary of Internet-related resources for a handful of fields related to numerical analysis.
• Object Oriented Numerics: Large collection of links to computing libraries
• Stony Brook Algorithm Repository: A comprehensive collection of algorithm implementations for over seventy of the most fundamental problems in combinatorial algorithms. The problem taxonomy, implementations, and supporting material are all drawn from Steve Skiena's excellent book The Algorithm Design Manual.
• TOMS Resource List: Links to peer-reviewed software, other software repositories and other resources, maintained by Transactions on Mathematical Software.
• UNESCO Free software Directory
• UTK Math Archive Links A very long list of links maintained by the University of Tennesee, Knoxville (which also writes their own software).