What is in your Mathematica tool bag?
We all know that Mathematica is great, but it also often lacks critical functionality. What kind of external packages / tools / resources do you use with Mathematica?
I'll edit (and invite anyone else to do so too) this main post to include resources which are focused on general applicability in scientific research and which as many people as possible will find useful. Feel free to contribute anything, even small code snippets (as I did below for a timing routine).
Also, undocumented and useful features in Mathematica 7 and beyond you found yourself, or dug up from some paper/site are most welcome.
Please include a short description or comment on why something is great or what utility it provides. If you link to books on Amazon with affiliate links please mention it, eg, by putting your name after the link.
Packages:
LevelScheme is a package that greatly expands Mathematica's capability to produce good looking plots. I use it if not for anything else then for the much, much improved control over frame/axes ticks. Its newest version is called SciDraw, and it will be released sometime this year. Presentation Package (US$50 - no charge for updates) grassmannOps package provides resources for doing algebra and calculus with Grassmann variables and operators that have non trivial commutation relations. GrassmannAlgebra package and book for working with Grassmann and Clifford algebras. Tools:
MASH is Daniel Reeves's excellent Perl script essentially providing scripting support for Mathematica v7. (Now built in as of Mathematica 8 with the -script option.) alternate Mathematica shell with a GNU readline input (using python, *nix only) Resources:
Wolfram's own repository MathSource has a lot of useful if narrow notebooks for various applications. Also check out the other sections such as
Current Documentation , Courseware for lectures, Demos for, well, demos. The Mathematica Wikibook.
Books:
web , pdf ) is a must read if you want to do anything more than For loops in Mathematica. We have the pleasure of having Leonid himself answering questions here. web ) Demonstrations Page . pdf ) - A good concise introduction to most of what you need to know about Mathematica programming. Undocumented (or scarcely documented) features:
this question . this answer this question . this question . I've mentioned this before, but the tool I find most useful is an application of Reap and Sow which mimics/extends the behavior of GatherBy :
SelectEquivalents[x_List,f_:Identity, g_:Identity, h_:(#2&)]:=
Reap[Sow[g[#],{f[#]}]&/@x, _, h][[2]];
This allows me to group lists by any criteria and transform them in the process. The way it works is that a criteria function ( f ) tags each item in the list, each item is then transformed by a second supplied function ( g ), and the specific output is controlled by a third function ( h ). The function h accepts two arguments: a tag and a list of the collected items that have that tag. The items retain their original order, so if you set h = #1& then you get an unsorted Union , like in the examples for Reap . But, it can be used for secondary processing.
As an example of its utility, I've been working with Wannier90 which outputs the spatially dependent Hamiltonian into a file where each line is a different element in the matrix, as follows
rx ry rz i j Re[Hij] Im[Hij]
To turn that list into a set of matrices, I gathered up all sublists that contain the same coordinate, turned the element information into a rule (ie {i,j}-> Re[Hij]+I Im[Hij]), and then turned the collected rules into a SparseArray all with the one liner:
SelectEquivalents[hamlst,
#[[;; 3]] &,
#[[{4, 5}]] -> (Complex @@ #[[6 ;;]]) &,
{#1, SparseArray[#2]} &]
Honestly, this is my Swiss Army Knife, and it makes complex things very simple. Most of my other tools are somewhat domain specific, so I'll probably not post them. However, most, if not all, of them reference SelectEquivalents .
Edit : it doesn't completely mimic GatherBy in that it cannot group multiple levels of the expression as simply as GatherBy can. However, Map works just fine for most of what I need.
Example : @Yaroslav Bulatov has asked for a self-contained example. Here's one from my research that has been greatly simplified. So, let's say we have a set of points in a plane
In[1] := pts = {{-1, -1, 0}, {-1, 0, 0}, {-1, 1, 0}, {0, -1, 0}, {0, 0, 0},
{0, 1, 0}, {1, -1, 0}, {1, 0, 0}, {1, 1, 0}}
and we'd like to reduce the number of points by a set of symmetry operations. (For the curious, we are generating the little group of each point.) For this example, let's use a four fold rotation axis about the z-axis
In[2] := rots = RotationTransform[#, {0, 0, 1}] & /@ (Pi/2 Range[0, 3]);
Using SelectEquivalents we can group the points that produce the same set of images under these operations, ie they're equivalent, using the following
In[3] := SelectEquivalents[ pts, Union[Through[rots[#] ] ]& ] (*<-- Note Union*)
Out[3]:= {{{-1, -1, 0}, {-1, 1, 0}, {1, -1, 0}, {1, 1, 0}},
{{-1, 0, 0}, {0, -1, 0}, {0, 1, 0}, {1, 0, 0}},
{{0,0,0}}}
which produces 3 sublists containing the equivalent points. (Note, Union is absolutely vital here as it ensures that the same image is produced by each point. Originally, I used Sort , but if a point lies on a symmetry axis, it is invariant under the rotation about that axis giving an extra image of itself. So, Union eliminates these extra images. Also, GatherBy would produce the same result.) In this case, the points are already in a form that I will use, but I only need a representative point from each grouping and I'd like a count of the equivalent points. Since, I don't need to transform each point, I use the Identity function in the second position. For the third function, we need to be careful. The first argument passed to it will be the images of the points under the rotations which for the point {0,0,0} is a list of four identical elements, and using it would throw off the count. However, the second argument is just a list of all the elements that have that tag, so it will only contain {0,0,0} . In code,
In[4] := SelectEquivalents[pts,
Union[Through[rots[#]]]&, #&, {#2[[1]], Length[#2]}& ]
Out[4]:= {{{-1, -1, 0}, 4}, {{-1, 0, 0}, 4}, {{0, 0, 0}, 1}}
Note, this last step can just as easily be accomplished by
In[5] := {#[[1]], Length[#]}& /@ Out[3]
But, it is easy with this and the less complete example above to see how very complex transformations are possible with a minimum of code.
Todd Gayley (Wolfram Research) just send me a nice hack which allows to "wrap" built-in functions with arbitrary code. I feel that I have to share this useful instrument. The following is Todd's answer on my question .
A bit of interesting (?) history: That style of hack for "wrapping" a built-in function was invented around 1994 by Robby Villegas and I, ironically for the function Message, in a package called ErrorHelp that I wrote for the Mathematica Journal back then. It has been used many times, by many people, since then. It's a bit of an insider's trick, but I think it's fair to say that it has become the canonical way of injecting your own code into the definition of a built-in function. It gets the job done nicely. You can, of course, put the $inMsg variable into any private context you wish.
Unprotect[Message];
Message[args___] := Block[{$inMsg = True, result},
"some code here";
result = Message[args];
"some code here";
result] /; ! TrueQ[$inMsg]
Protect[Message];