As with last time out, I’ll illustrate how orditorp() works via a PCA biplot for the Dutch dune meadow data.

## load vegan and the data
require(vegan)
data(dune)
ord <- rda(dune) ## PCA of Dune data

## species priority; which species drawn last, i.e. on top
priSpp <- diversity(dune, index = "invsimpson", MARGIN = 2)
## sample priority
priSite <- diversity(dune, index = "invsimpson", MARGIN = 1)

## scaling to use
scl <- 3

I won’t explain any of the code above; it is the same as that used in the earlier post where an explanation was also provided.

orditorp() takes an ordination object as the first argument and in addition the display argument controls which set of scores is displayed. Note that orditorp() can only plot one set of scores at a time, which as we’ll see in a minute is not exactly ideal nor foolproof. Like ordilabel(), you are free to specify the importance of each sample or species via argument priority. In ordilable() the priority controlled the plotting order such that those samples or species with high priority were plotted last (uppermost). Instead, orditorp() draws labels for samples or species (if it can) for those with the highest priority first.

So we have something to talk to, recreate the basic samples and species biplot as used in the previous post but updated to use orditorp()

plot(ord, type = "n", scaling = 3)
orditorp(ord, display = "sites", priority = priSite, scaling = scl,
         col = "blue", cex = 1, pch = 19)
## You may prefer separate plots, but here species as well
orditorp(ord, display = "species", priority = priSpp, scaling = scl,
         col = "forestgreen", pch = 2, cex = 1)

PCA biplot of the Dutch dune meadow data produced using orditorp()

The behaviour or orditorp() should now be reasonably clear; labels are drawn for sample or species only if there is room to do so, with a point being used instead. orditorp() isn’t perfect by any means. Because it can only drawn one set of scores at a time, there is no easy way to stop the species labels plotting over the sample labels and vice versa.

How it works is, first orditorp() calculates the heights and widths of the labels, adds a bit of space to this (more on this later) and then works out if the box given by the current sample or species label width/height, centred on the axis score coordinate, will obscure the label boxes of any labels previously drawn. If the label box doesn’t obscure any previous label boxes the label is drawn at the sample or species score coordinates. If it does obscure an existing label then a point is drawn instead. orditorp() draws the labels in order of priority and as it draws each subsequent label it checks to see if previous labels are not obscured.

This process isn’t infallible of course; for example the second highest priority sample or species could lie very close to the highest priority one in ordination space and if so orditorp() would not draw a label for this second highest priority sample or species because it would obscure the label of the highest priority one.

The amount of spacing or padding around each label is specified via the air argument which has a default of 1. air is interpreted as the proportion of half the label width or height that the label occupies. The default of 1 therefore means that in fact there is no additional spacing beyond the confines of the box that encloses the label. If air is greater than 1 proportionally more padding is added whilst values less than 1 indicate that labels can overlap. The figure below shows the species scores only with two values for air. In the left hand panel air = 2 is used and the labels are padded either side of the label by the entire string width or height. The right hand panel uses air = 0.5 which allows labels to overlap by up to a quarter of the string width or height in any direction from the plotting coordinate (in other words, the box that cannot be obscured when plotting subsequent labels is half the string width wide and half the string height high, centred on the plotting coordinates for the label).

layout(matrix(1:2, ncol = 2))
op <- par(mar = c(5,4,4,1) + 0.1)
## site/sample scores
plot(ord, type = "n", scaling = 3, main = expression(air == 2), cex = 1)
orditorp(ord, display = "species", priority = priSite, scaling = scl,
         col = "forestgreen", cex = 1, pch = 2, air = 2)
## Species scores
plot(ord, type = "n", scaling = 3, main = expression(air == 0.5), cex = 1)
orditorp(ord, display = "species", priority = priSpp, scaling = scl,
         col = "forestgreen", pch = 2, cex = 1, air = 0.5)
par(op)
layout(1)

PCA species plot of the Dutch dune meadow data produced using orditorp() showing the effect of changing argument air.

One additional point to make is that orditorp() returns a logical vector indicating which sample or species scores were drawn with labels (TRUE) or points (FALSE), which might be useful for further plotting or adding to the diagram.

So there were have orditorp(). Next time I’ll take a look at ordipointlabel() which tackles the problem of producing a tidy ordination diagram in a far more complex way than either ordilabel() or orditorp().

First up we have ordilabel().

One of the problems that ordination results pose is that there is a lot is a lot information that we want to convey using a relatively small number of pixels. What we often end up with is a jumbled mess and because of the way the sample or species scores are plotted, the important observations could very well end up covered in all the rare species or odd samples just by virtue of their ordering in the data set.

The simplest tool that vegan provides to help in this regard is ordilabel(); it won’t produce a publication-ready, uncluttered ordination diagram but it will help you focus on the “important”¹ things.

ordilabel() draws sample or species scores with their label (site ID or species name/code) taken from the dimnames of the data used to fit the ordination. To help their display, however, ordilabel() draws the labels in a box with an opaque background so that the labels plotted later (i.e. above) cover earlier labels whilst remain visible because of the opaque background. ordilabel() also allows you to specify the importance of the samples or species via the priority argument, which in effect controls which labels get drawn first or beneath all the others.

Here I’ll use a PCA of the famous Dune Meadow data². First, we load vegan and the data and perform the ordination

require(vegan)
data(dune)
ord <- rda(dune) ## PCA of Dune data

In this example, I want to give plotting priority to those species or samples that are most abundant or most diverse, respectively. For this I will use Hill’s N₂ for both the species and the samples, both of which can be computed via the diversity() function

## species priority; which species drawn last, i.e. on top
priSpp <- diversity(dune, index = "invsimpson", MARGIN = 2)
## sample priority
priSite <- diversity(dune, index = "invsimpson", MARGIN = 1)

The MARGIN argument refers to which dimension or margin of the data is used; 1 means rows, 2 means columns. Hill’s N₂ is equal to the inverse (or reciprocal) of the Simpson diversity measure.

Throughout I’m going to use symmetric scaling of the two sets of scores for use in the biplot. As it is important to make sure the same scaling is used at each stage it is handy to store the scaling in an object and then refer to that object throughout. That way you can easily change the scaling used by altering the value in the object. Here I use scl and symmetric scaling is indicated by the number 3

## scaling to use
scl <- 3

ordilabel() adds labels to an existing plot, so first set up the plotting region for the PCA biplot using the plot() method with type = "n" to not plot any of the data

plot(ord, type = "n", scaling = 3)

Now we are ready to add labels to the plot. ordilabel() takes the ordination object as the first argument and extracts the scores indicated by the display argument from the fitted object. There are a number of standard plotting arguments to control the look and feel of the labels, but the important argument is priority to control the plotting order. Here we set it to the Hill’s N₂ values we computed earlier. The code chunk below adds both to the base plot we just generated

ordilabel(ord, display = "sites", font = 3, fill = "hotpink",
          col = "blue", priority = priSite, scaling = scl)
## You may prefer separate plots, but here add species as well
ordilabel(ord, display = "species", font = 2,
          priority = priSpp, scaling = scl)

The resulting biplot should look similar to the one below

PCA biplot of the dune meadow data with labels added by ordilabel()

Not perfect, but better than the standard plot() method in vegan.

Alternatively, one might wish to draw side by side biplots of the sample and species scores. This can be done simply with a call to layout() to split the current plot device into two plot regions, which we fill using very similar plotting commands as described above

layout(matrix(1:2, ncol = 2))
plot(ord, type = "n", scaling = scl)
ordilabel(ord, display = "sites", font = 3, fill = "hotpink",
          col = "blue", priority = priSite, scaling = scl)
plot(ord, type = "n", scaling = scl)
ordilabel(ord, display = "species", font = 2,
          priority = priSpp, scaling = scl)
layout(1)

Side-by-side PCA biplots of the dune meadow data with labels added by ordilabel()

You may notice some warnings about scaling not being a graphical parameter. These are harmful and arise because we pass scaling along as part of the ... argument which we also pass on to the plotting functions used to build the plot. We’ve tried hard to stop these warnings in vegan using a technique I blogged about a while back, but it looks like we missed a few of these. It will be fixed in a later version of vegan and the warnings will go away.

Next time we’ll look at orditorp().

Notes:
¹Whatever “important” means…
²Not that I think this is the best way to analyse these data, it is just for show!

The problem is how to shade or colour in areas under a curve. The more general problem is how to do this when you don’t have any data that fall on the margins of the regions you wish to shade. Here is more solution to that more general problem.

As I don’t have Brian’s data lets generate some similar data with the help of the mgcv package

library(mgcv)
set.seed(2) ## simulate some data... 
dat <- gamSim(1, n = 400, dist = "normal", scale = 2)
b <- gam(y ~ s(x2), data = dat)
set.seed(42)
newX <- with(dat, data.frame(x2 = sort(runif(100, min = min(x2), max = max(x2)))))
pred <- predict(b, newdata = newX)

bed <- data.frame(Volume = pred, Diameter = newX[,1])

I’m not going to explain that as it is but a means to an end. The resulting data can be nicely, plotted via

plot(Volume ~ Diameter, data = bed, type = "o", pch = 19)

with the resulting plot shown in Figure 1, below.

Figure 1: Example data used to illustrate shading areas under a curve

To illustrate, let’s assume we want to shade the regions under the curve as defined by the following start and end points of four regions

from <- c(0.1, 0.25, 0.37, 0.78)
to <- c(0.25, 0.37, 0.63, 0.84)

To cut to the chase, here is my solution to the problem:

polyCurve <- function(x, y, from, to, n = 50, miny,
                      col = "red", border = col) {
    drawPoly <- function(fun, from, to, n = 50, miny, col, border) {
        Sq <- seq(from = from, to = to, length = n)
        polygon(x = c(Sq[1], Sq, Sq[n]),
                y = c(miny, fun(Sq), miny),
                col = col, border = border)
    }
    lf <- length(from)
    stopifnot(identical(lf, length(to)))
    if(length(col) != lf)
        col <- rep(col, length.out = lf)
    if(length(border) != lf)
        border <- rep(border, length.out = lf)
    if(missing(miny))
        miny <- min(y)
    interp <- approxfun(x = x, y = y)
    mapply(drawPoly, from = from, to = to, col = col, border = border,
           MoreArgs = list(fun = interp, n = n, miny = miny))
    invisible()
}

Don’t worry, I’ll explain all of that in a minute; first let’s see polyCurve() in use

cols <- c("red", "forestgreen", "navyblue", "orange")
with(bed, plot(Diameter, Volume, type = "o", pch = 19,
               panel.first =
               polyCurve(Diameter, Volume, from = from, to = to,
                         col = cols, border = "black")))

The resulting plot should look like the one in Figure 2 below.

Figure 2: Final plot showing the example data and four regions under the curve shaded

Now the nitty gritty. polyCurve() takes the x and y data points, the start and end points of the areas to be shaded in (arguments from and to), an option to override the minimum value on the y axis to which the shading will extend (can be missing), plus vectors of colours for the fill and border of each polygon drawn.

Lines 3–8 define an internal function drawPoly(), which will actually draw a single polygon over the region under the curve defined by its arguments from and to. The first argument to drawPoly() is fun, which is a function that returns values on the curve for a vector of locations in the x variable/axis. We’ll see how this function is derived later. Notice that we pass in here some of the arguments previously described that control the look of the polygons and how far down on the y-axis the polygon will be drawn (miny).

The first line of drawPoly() generates an equally-spaced sequence of n values over the range of the polygon on the x-axis. n defaults to 50 values but this can be changed if needed especially if the data in that region are very wiggly or the region quite wide. The next part of drawPoly() uses the polygon() function to actually draw the polygon. We pass it the x coordinate as the sequence of values just created, with the first and last points in the sequence repeated. The y coordinates are supplied as the output from fun() for the sequence of points, augmented at the start and end by the value of miny. And that’s it.

Lines 9–16 of polyCurve() do some sanity checking and house keeping

• First the lengths of from and to are checked to see if they are equal
• Then we check the lengths of the vectors of fill and border colours, col and border to see if these match up with the number of polygons to be drawn. If the lengths don’t match, we extend each vector to match the number of polygons to be drawn. This is a nice little feature that allows for a single colour to be supplied and have polyCurve() still work.
• Finally, we check to see if argument miny was set by the user and if not we assign to it the minimum value taken by y.

The next line is where some R magic happens. Recall that drawPoly() takes a function as its first argument, which returns interpolated values along the data curve at specified x variable locations. This is where that function is created. approxfun() is one of those great little R functions that really saves a lot of time and coding. Essentially, approxfun() linearly (by default) interpolates a set of x and y coordinates. But crucially, and here is the kicker, it returns a function that, if given new locations for the x coordinate, will return interpolated values for the y coordinate.

Rather than interpolating the data curve for each region for which we want to draw a polygon, we interpolate the entire data curve with approxfun() and then reuse that function to generate the interpolated values we need when drawing each region’s polygon.

The last major piece of polyCurve() code is where we repeatedly call drawPoly(), once per region to be covered by a polygon. I could have done this part with a for() loop, iterating over the regions and calling drawPoly() with the appropriate from, to, col and border, etc. That would be relatively easy to do, but it is not really R-like. R provides a family of functions, known as the apply functions, which in many cases allow one to do away with an explicit for(). [Note that the loop is still there, it is just hidden away from view and in some cases done in compiled code rather than interpretted R code.]

We want to call drawPoly() for each combination of from, to, col and border. For this we need a the multivariate apply function mapply(). We pass mapply() the function we wish to repeatedly call. After this we give the arguments we wish to call our function with. mapply() will call our function once for each combination of these arguments. I.e. it will call drawPoly() with from[i], to[i], col[i] and border[i], where i takes the value 1, 2, 3, … in turn. The final part of our mapply() call is to pass some extra arguments needed for drawPoly(); these arguments don’t change with each polygon so they are supplied as a list object to the MoreArgs argument. Notice that I name the elements of this list using the name of the drawPoly() argument I want each element passed on to.

The final line is last bit of house keeping; polyCurve() returns nothing and does so invisibly.

Let’s return to the code we used to actually draw Figure 2 above

cols <- c("red", "forestgreen", "navyblue", "orange")
with(bed, plot(Diameter, Volume, type = "o", pch = 19,
               panel.first =
               polyCurve(Diameter, Volume, from = from, to = to,
                         col = cols, border = "black")))

This is a fairly standard call to plot(). The none-standard part is the use of the panel.first argument. This is actually an argument of plot.default(), the default plot method. It takes an R expression, a bit of R code, that will be run after the plotting region has been defined and axes drawn, but crucially before the data for the plot are actually drawn. This is where we want polyCurve() to be run, so the coloured polygons end up being drawn underneath the actual data. This produces a nicer looking plot than having the polygons drawn over the top of the data.

It is worth noting that there is a corresponding panel.last argument which works the same way but is only run once all the other plotting is complete. A further point to note is that these two arguments work nicely when the default plot is called, but they can break when other plot methods are called first. Things break because the expression supplied to panel.first might end up getting evaluated (run) before any plotting has even taken place, because the argument is being evaluated in the wrong place (at the wrong time). At the very least, panel.first will have no effect, but it might raise an error in some situations.

So there we have it. Interpolating the data allows for a relatively concise solution to the problem of shading areas under a curve. It is a general solution not requiring one to have data at the boundaries of the regions to be shaded and as such doesn’t require any selection of data points within the region to draw the polygon through.

If you are still with me, it might be useful to visualise how drawPoly() and polyCurve() work, to see what each part of the process is doing.

First, set up a base plot onto which we can draw; this shows the data as before, but with the data points draw in a smaller size.

plot(Volume ~ Diameter, data = bed, type = "o", pch = 19, col = "black",
     cex = 0.5, main = "Interpolated points on the\ndata curve")

Next, use approxfun() to produce an interpolation function for the data

FUN <- with(bed, approxfun(Diameter, Volume))

FUN() takes a single argument, the locations on the x variable for which interpolated y coordinates are to be returned, e.g.

> FUN((1:10) * 0.1) 
 [1]  8.394756 12.740232 12.004102  8.834530  7.239627  8.145023
 [7]  7.831734  5.340844  4.948277        NA

Notice that NA is returned for values outside the range of the data; This is the default behaviour of approxfun(), which can be changed via argument rule, but we can’t get it to extrapolate beyond the range of the data.

Now, generate a set of x coordinates for the region of the curve we want to interpolate. Here I use the bit of the curve between the two peaks in the data.

Sq <- seq(from[3], to[3], length = 20)

We use 20 values here, so the plot we will produce in a minute isn’t overly crowded, but the more values you draw over the region, the smoother the fit to the data curve itself.

The interpolated values for this sequence of coordinates is given by FUN()

> FUN(Sq)
 [1] 9.724055 9.300774 8.907145 8.565961 8.233448 7.924078
 [7] 7.675439 7.471385 7.315091 7.263203 7.216052 7.210351
[13] 7.339273 7.468195 7.600860 7.801945 7.996272 8.180440
[19] 8.347901 8.429620

and we can draw these locations on the plot via a call to points(), giving it the x coordinates, Sq, and the output from FUN(Sq)

points(Sq, FUN(Sq), col = "#FF000088", pch = 19, type = "o")

The points were drawn in red, with some alpha transparency so that the data and curve show through from underneath. The resulting plot should look like the one in the left hand panel of Figure 3 below.

Figure 3: Illustrating the steps involved in interpolating the data curve and drawing a polygon under the curve

Now that we have a good handle on what approxfun() is doing, we can move on to the drawing of the polygon that will shade in the area under the curve defined by our region. Start a new plot as before

plot(Volume ~ Diameter, data = bed, type = "o", pch = 19, col = "black",
     cex = 0.5, main = "The final polygon")
FUN <- with(bed, approxfun(Diameter, Volume))
Sq <- seq(from[3], to[3], length = 20)

We now need to do a few housekeeping steps that will make the subsequent plotting much easier.

miny <- with(bed, min(Volume))
xvals <- c(Sq[1], Sq, Sq[20])
yvals <- c(miny, FUN(Sq), miny)
col <- "#FF000088"

First we looked-up the minimum value of the y coordinate, Volume. Then we created a set of x and y coordinates for which we want the polygon drawing. For the x coordinates, notice how we extend the sequence by prepending the first element of Sq and appending the last element on to the vector of x coordinates Sq. We do this because we have two points with the same x coordinate at the edges of the region we want to cover in a polygon; one on the curve and one at the bottom of the plot. The y coordinates were generated by calling our interpolation function FUN(), and as with the x coordinates, we pad this vector of coordinates at both ends with the minimum value of Volume. This takes care of the vertices of the polygon that fall to the bottom of the plot. We also store the plotting colour so we don’t have to keep repeating it in the steps to follow.

Having done that bit of housekeeping, we can draw the polygon. In the code below I draw the actual polygon and overlay on it the vertices of the polygon through which R actually draws the line of the polygon

polygon(xvals, yvals, border = col)
points(xvals, yvals, col = col, pch = 19, type = "o")

(Note that the behaviour of polygon() is to join the first and last vertices, hence we didn’t need to do that bit ourselves.)

At this point the plot should look like the right hand panel of Figure 3, above.

The entirety of Figure 3 can be reproduced via the following code

layout(matrix(1:2, ncol = 2))
op <- par(mar = c(5,4,4,2) + 0.1)
## plot1
plot(Volume ~ Diameter, data = bed, type = "o", pch = 19, col = "black",
     cex = 0.5, main = "Interpolated points on the\ndata curve")
FUN <- with(bed, approxfun(Diameter, Volume))
Sq <- seq(from[3], to[3], length = 20)
points(Sq, FUN(Sq), col = "#FF000088", pch = 19, type = "o")
## plot2
plot(Volume ~ Diameter, data = bed, type = "o", pch = 19, col = "black",
     cex = 0.5, main = "The final polygon")
miny <- with(bed, min(Volume))
xvals <- c(Sq[1], Sq, Sq[20])
yvals <- c(miny, FUN(Sq), miny)
col <- "#FF000088"
polygon(xvals, yvals, border = col)
points(xvals, yvals, col = col, pch = 19, type = "o")
par(op)
layout(1)

In simple weighted averaging (WA) there is little to specify other than the predictors (the species or other proxy data) and the response (the thing you wish to build a model for and predict). The one user-specified option in a simple WA is the type of deshrinking to use.

Why deshrinking? Well, in a WA model, averages of the response are effectively taken twice; i) first when the WA optima of the response variable for each taxon is computed, and ii) a second time when a weighted average of the optima for the species in each sample is computed to give the raw WA estimate of the response for each sample. Note that the weights here are the values in the predictor data matrix; usually these are species or taxon abundances (often the in the form of proportions or percentages).

The effect of taking averages twice is to shrink the range of possible estimates from a WA model. This can be illustrated using some of the tools from analogue. First load the package and the SWAP example data set

## load analogue...
require(analogue)
## ...and some example data
data(swapdiat)
data(swappH)

Next, compute the WA pH optima for each diatom taxon

opt <- optima(swapdiat, swappH) ## compute WA optima

The final step is to use some simple matrix algebra to give the raw WA estimates of the pH for each SWAP sample (site) [note that the swapdiat object needs to be cast as a matrix for the matrix algebra step, which we do using data.matrix()]

diat <- data.matrix(swapdiat)  ## convert to a matrix
pred <- ((diat %*% opt) / rowSums(diat))[, 1] ## compute raw WA estimates

I won’t go into the details of the last step; it is a reasonably optimised bit of R code to compute a weighted average of the pH optima for each species in a given sample. [The first part of the code involving the matrix multiplication operator %*% forms a weighted sum of the pH optima, with weights given by the abundances of the diatom taxa.]

From the above we can see that there are three types of pH value:

• the observed pH in swappH
• the pH optima for each taxon in opt, and
• the WA estimate of pH for each site in pred.

A quick look at the range of the pH values for each stage shows the shrinkage problem resulting from averaging

r.obs <- range(swappH)
r.opt <- range(opt)
r.wa <- range(pred)

> rbind(r.obs, r.opt, r.wa)
          [,1]    [,2]
r.obs 4.330000 7.25000
r.opt 4.500000 7.25000
r.wa  4.875662 6.62395

About a whole pH unit has been lost due to the repeated taking of averages.

Clearly, this is a problem for WA models, one that is addressed through the using a deshrinking step.

Traditionally there were two approaches to deshrinking; inverse and classical deshrinking. In the inverse approach, a model of the form

,

which is a simple linear regression with the observed values as the response and the raw WA estimates as the predictor. The classical deshrinking approach flips the role of the response and the predictor such the model is

.

Expanded WA estimates are achieved by taking the usual predicted values from the inverse model or by inverting the classical model equation.

Both the inverse and classical deshrinking approaches are linear. The idea of using a non-linear deshrinking step has long been proposed in the literature, all the way back to ter Braak and Juggins (1993) and Marchetto (1994), but in practice the idea has not caught on, presumably because of a lack of widely-available and user-friendly software that implement the technique.

In the non-linear deshrinking approach the following model is use, which very similar to that of the inverse approach



where is a smooth, monotonic function. The monotonicity constraint is important; as the raw WA estimate increases in value the expanded estimate should also increase. In other words we don’t allow for the expanded estimate to get smaller (bigger) as the raw WA estimate gets bigger (smaller).

New code in analogue implements this monotonic deshrinking idea. The seed of the actual implementation used here is murky.¹ Briefly, both analogue and rioja fit a cubic regression spline via s(wa.est, bs = "cr"), but as the standard gam() function won’t constrain the smooth to be monotonic some additional steps have to be performed, and we end up fitting a penalised regression with monotonicity constraints invoked. The code to show how to do this in mgcv was sitting there in one of Simon’s man pages!

So what does this look like? The figure below shows the raw WA pH estimates and the observed pH values for the SWAP diatom data we looked at earlier. The thick green line fitted through the points is the monotonic cubic regression spline fitted via mgcv.




Figure 1: Monotonic deshrinking spline for the SWAP diatom pH data set


The monotonic deshrinking and the plot were produced using

## do the monotonic deshrinking
mono <- deshrink(swappH, pred, type = "monotonic")$env

## need to get things in increasing order for plotting
ord <- order(pred)

## draw the data and the fitted monotonic cubic regression spline
plot(pred, swappH, asp = 1, ylim = r.obs, xlim = r.obs,
     panel.first = abline(0, 1, col = "darkgrey", lwd = 2),
     ylab = "Observed pH", xlab = "Raw WA pH")
lines(mono[ord] ~ pred[ord], type = "l", col = "forestgreen", lwd = 2)

In this example, there appears to be two relationships between the raw WA estimates and the observed pH; a steeper one up to pH 6.0 and a somewhat less strong one thereafter.

Figure 2 shows a comparison of the three deshrinking methods discussed above.




Figure 2: Comparison of inverse, classical and monotonic deshrinking for the SWAP diatom pH data set


The inverse and classicial deshrinking values were computed using

inv <- deshrink(swappH, pred, type = "inverse")$env
cla <- deshrink(swappH, pred, type = "classical")$env

The figure was produced using

ylim <- range(r.obs, r.wa, mono, inv, cla)
plot(pred, swappH, asp = 1, ylim = ylim, xlim = ylim,
     panel.first = abline(0, 1, col = "darkgrey", lwd = 2),
     ylab = "Observed pH", xlab = "Raw WA pH")
ord <- order(pred)
lines(inv[ord] ~ pred[ord], type = "l", col = "darkorange", lwd = 2)
lines(cla[ord] ~ pred[ord], type = "l", col = "navyblue", lwd = 2)
lines(mono[ord] ~ pred[ord], type = "l", col = "forestgreen", lwd = 2)
legend("topleft", legend = c("Monotonic", "Inverse", "Classical"),
       col = c("forestgreen","darkorange","navyblue"),
       lwd = 3, bty = "n")

The monotonic deshrinking curve is quite similar to the inverse deshrinking one; this is not surprising as monotonic deshrinking is a local version of the inverse deshrinking model. The two curves only deviate from one another above pH 5.5.

There does seem to be a slight improvement in WA model performance when using monotonic deshrinking over the other deshrinking techniques, for the SWAP data set at least. Well, that was the conclusion of the paper John and I have been working on for a special issue of JoPL. But you’ll have to wait for the paper (and accompanying blog post) for the full details when it is accepted and in press.

Notes
¹Steve Juggins discussed the idea briefly in a presentation he gave at UCL a number of years ago and I had at the back of my mind been mulling how to do this in R without having code the entire thing(!) myself. John Birks recently suggested that I use monotonic deshrinking in a comparison of transfer function methods we were doing for a special issue of the Journal of Paleolimnology (JoPL). It turns out that Richard Telford, a colleague of John’s, had discussed this with both John and Steve too including the implementation used in analogue and, as it turned out, also in Steve’s rioja R package. It seems Steve and I implemented almost the same idea, independently; me after I’d been scouring Simon Wood’s exceedingly useful man pages for his mgcv package trying to find out how to do something with gam() to analyse time series data.

References
Marchetto (1994; Journal of Paleolimnology 12, 155–162)
ter Braak & Juggins (1993; Hydrobiologia, 269/270, 485–502)


]]> http://ucfagls.wordpress.com/2013/01/05/monotonic-deshrinking-in-weighted-averaging-models/feed/ 0 ucfagls Monotonic deshrinking spline for the SWAP diatom pH data set Comparison of inverse, classical and monotonic deshrinking for the SWAP diatom pH data set http://ucfagls.wordpress.com/2013/01/04/a-new-version-of-analogue-for-a-new-year/ http://ucfagls.wordpress.com/2013/01/04/a-new-version-of-analogue-for-a-new-year/#comments Fri, 04 Jan 2013 21:00:57 +0000 ucfagls http://ucfagls.wordpress.com/?p=569 Continue reading →]]> Yesterday I rolled up a new version (0.10-0) of analogue, my R package for analysing palaeoecological data. It is now available from CRAN.

There were lots of incremental changes to Stratiplot() to improve the quality of the stratigraphic diagrams produced and fix several annoying bugs. Also the definition of the standard error of MAT reconstructions was fixed; it is essentially a weighted variance but the original version assumed the weights summed to 1, which is not the case for dissimilarities of the k-NN.

Several new functions and additional functionality were added to the package:

• caterpillarPlot() produces caterpillar plots showing species WA optima and tolerance ranges
• splitSample() is a convenience function for sampling a subset of training set samples whilst ensuring that the entire environmental gradient of interest in the training set is evenly sampled.
• The wa() function received a lot of love in this iteration. The main addition is to allow non-linear deshrinking of the raw WA estimates alongside the more common inverse and classical deshrinking techniques. The deshrinking is achieved using a cubic regression spline fitted using the gam() function in package mgcv. The spline is constrained to be monotonic to make sure that the deshrunk values for increasing raw values are likewise increasing. Small tolerance handling in wa() with tolerance downweighting gained the option to replace small tolerances with the mean of all taxon tolerances.
• logitreg(), which applies a logistic regression to the problem of identifying a critical threshold in compositional dissimilarity for MAT models, saw a major update. The returned object was substantially altered to allow for a wider amount of information to be supplied to the user. fitted() and predict() methods for class "logitreg" were also added. These compute the fitted probabilities for the training set samples and for new (e.g. fossil) samples respectively. The probabilities are in respect to the analogue-ness of samples to the groups in the training set (e.g. vegetation biomes in the case of pollen data).
  These changes allow an analysis similar in spirit to that of Gavin et al (2003, Quaternary Research 60; 356–367) in their Figure 8. Here though logistic regression fits are used and not the ROC method they use.

I’ll be writing more on these ideas, especially the monotonic deshrinking and the logistic regression approach to dissimilarity threshold choice in future posts.


]]> http://ucfagls.wordpress.com/2013/01/04/a-new-version-of-analogue-for-a-new-year/feed/ 1 ucfagls http://ucfagls.wordpress.com/2012/08/15/processing-sample-labels-using-regular-expressions-in-r/ http://ucfagls.wordpress.com/2012/08/15/processing-sample-labels-using-regular-expressions-in-r/#comments Wed, 15 Aug 2012 09:49:29 +0000 ucfagls http://ucfagls.wordpress.com/?p=555 Continue reading →]]> I am often found in possession of palaeo core data where the sample identifiers contain a core code or label plus the sample depth. Often these are things generated by colleagues who have used other software where for one reason or another they don’t want to store the depth information as a separate numeric variable. I also generate such data sets, not because I want to but because the software often supplied with lab equipment (most recent example is the Thermo Flash EA/Delta V I’ve been running stable N and C isotope measurements on) that records data/measurements using a single character identifier variable.

The information in these labels is useful and I really don’t want to type out all the depths again and it’s not just because I am lazy; the more times you have to enter data the more opportunities for transcription errors to creep into your work and analysis. So I have things like this

R> (eg1 <- paste0("CORE", 0:10 + 0.5))
 [1] "CORE0.5"  "CORE1.5"  "CORE2.5"  "CORE3.5"  "CORE4.5"  "CORE5.5"
 [7] "CORE6.5"  "CORE7.5"  "CORE8.5"  "CORE9.5"  "CORE10.5"
R> (eg2 <- paste0("FOO_", 0:10 + 0.5))
 [1] "FOO_0.5"  "FOO_1.5"  "FOO_2.5"  "FOO_3.5"  "FOO_4.5"  "FOO_5.5"
 [7] "FOO_6.5"  "FOO_7.5"  "FOO_8.5"  "FOO_9.5"  "FOO_10.5"

What can be done to process these sorts of data with R to extract the useful information?

With eg2 we could split the strings on _ using strsplit() and process the resulting components. For example

R> as.numeric(sapply(strsplit(eg2, "_"), `[`, 2))
 [1]  0.5  1.5  2.5  3.5  4.5  5.5  6.5  7.5  8.5  9.5 10.5

To see how that code works, note that strsplit() returns a list with as many components as elements in the character vector supplied (e.g. length(eg2)). Each component of the list contains the individual character strings created by splitting.

R> head(spl <- strsplit(eg2, "_"), 2)
[[1]]
[1] "FOO" "0.5"

[[2]]
[1] "FOO" "1.5"

Notice that the depth information is in the second element of each list component. To access this information for the first component we might use spl[[1]][2] and the second one via spl[[2]][2]. Notice that the only thing that is changing here is the number in the [[ ]]. To each of the components of spl we are applying the [ function with argument 2; that can be automated via sapply() as shown above. The last part of the example just coerces the character vector of depths to a numeric one.

All of that is a bit of a faff and won't work for eg1 because there is nothing to split on. An alternative solution is to use regular expressions. I'm no regular expression expert and if there is anything in computing that will warp your feeble little mind it is a regular expression. However, these things are incredibly useful for matching or extracting bits of data from strings.

A regular expression contains placeholders or entities that you want to match or find within a given set of strings. For example, here is a modified version of eg1 where the last element has a different format to the rest

R> (eg3 <- c(eg1, "12.5CORE"))
 [1] "CORE0.5"  "CORE1.5"  "CORE2.5"  "CORE3.5"  "CORE4.5"  "CORE5.5"
 [7] "CORE6.5"  "CORE7.5"  "CORE8.5"  "CORE9.5"  "CORE10.5" "12.5CORE"

To match only those with one or more alphabetical characters are the start of the string we can use "^[A-Za-z]+" as our regular expression and the grep() to do the matching

R> grep("^[A-Za-z]+", eg3, value = TRUE)
 [1] "CORE0.5"  "CORE1.5"  "CORE2.5"  "CORE3.5"  "CORE4.5"  "CORE5.5"
 [7] "CORE6.5"  "CORE7.5"  "CORE8.5"  "CORE9.5"  "CORE10.5"

The [A-Za-z] means match anything that is a letter in the English language alphabet. I added a qualifier, the +, which means match one or more of these letters. The last bit of the regular expression is the ^, which indicates that matches should begin with one or more letters; anything that doesn't begin with one or more letters will not be matched. If you look carefully at the result, "12.5CORE" is missing because it doesn't start with one or more letters.
To match one or more letters at the end of a string, the $ can be used, e.g.

R> grep("[A-Za-z]+$", eg3, value = TRUE)
[1] "12.5CORE"

Let's turn our attention back to eg1. A regular expression that would match each component of the strings could be "([A-Za-z]+)([0-9\\.]+)". The parentheses group the various parts of the expression which we'll use in a moment. The first set of parentheses matches one or more letters whilst the second set matches one or more digits plus the decimal point. The decimal point has been escaped (which in R requires two not the usual one backslash) as it is a regular expression meta character (like + and *) that matches a single character. We want a literal . so we escape its usual meaning. As we now have a regular expression that will match the format of our sample labels we can proceed to manipulate them. This is where the parentheses come in. As I said, these group matches within the single expression. The matches within the parentheses can be referred to using backreferences. So I could use \\1 to refer to the strings matched by the first set of parentheses and \\2 to matches in the second set. Note we need to double backslash here as this is R.

To achieve our final goal of extracting the depth information from the sample labels we can combine this regular expression with the gsub() function, which does string replacement using regular expressions. If we think about what we want to do, we want to essentially replace the sample label with the extracted depth information to form a new set of strings. So we can match the two parts of our sample labels using our regular expression and replace them with a backreference to the depth part matched by the second set of parentheses. For example:

R> gsub("([A-Za-z]+)([0-9\\.]+)", "\\2", eg1)
 [1] "0.5"  "1.5"  "2.5"  "3.5"  "4.5"  "5.5"  "6.5"  "7.5"  "8.5"  "9.5"
[11] "10.5"

All that remains is to coerce that to numeric and we have our depth data

R> as.numeric(gsub("([A-Za-z]+)([0-9\\.]+)", "\\2", eg1))
 [1]  0.5  1.5  2.5  3.5  4.5  5.5  6.5  7.5  8.5  9.5 10.5

eg2 can be handled in a similar way but we need to add _ to the characters matched by the first set of parentheses

R> as.numeric(gsub("([A-Za-z_]+)([0-9\\.]+)", "\\2", eg2))
 [1]  0.5  1.5  2.5  3.5  4.5  5.5  6.5  7.5  8.5  9.5 10.5

or add it as a literal _ between the two sets

R> as.numeric(gsub("([A-Za-z]+)_([0-9\\.]+)", "\\2", eg2))
 [1]  0.5  1.5  2.5  3.5  4.5  5.5  6.5  7.5  8.5  9.5 10.5

If you had a more complicated data set with several cores in the same file, identified by a different core code, regular expressions can be used to extract the core and depth information. For example, given

R> set.seed(1)
R> dat <- data.frame(Label = paste0(rep(c("WAST", "NAGA"), each = 3), rep(0:2 + 0.5, 3)),
+                    Value = runif(6))
R> dat
    Label     Value
1 WAST0.5 0.2655087
2 WAST1.5 0.3721239
3 WAST2.5 0.5728534
4 NAGA0.5 0.9082078
5 NAGA1.5 0.2016819
6 NAGA2.5 0.8983897

we could add site and label data using

R> rexp <- "([A-Za-z]+)([0-9\\.]+)"
R> dat <- transform(dat, Site  = gsub(rexp, "\\1", Label),
+                        Depth = as.numeric(gsub(rexp, "\\2", Label)))
R> dat
    Label     Value Site Depth
1 WAST0.5 0.2655087 WAST   0.5
2 WAST1.5 0.3721239 WAST   1.5
3 WAST2.5 0.5728534 WAST   2.5
4 NAGA0.5 0.9082078 NAGA   0.5
5 NAGA1.5 0.2016819 NAGA   1.5
6 NAGA2.5 0.8983897 NAGA   2.5

These are just some very simple regular expressions but hopefully you can see their power and utility for manipulations of character data that palaeo-types often have to handle.


]]> http://ucfagls.wordpress.com/2012/08/15/processing-sample-labels-using-regular-expressions-in-r/feed/ 1 ucfagls http://ucfagls.wordpress.com/2012/07/24/whats-wrong-with-loess-for-palaeo-data/ http://ucfagls.wordpress.com/2012/07/24/whats-wrong-with-loess-for-palaeo-data/#comments Tue, 24 Jul 2012 10:00:36 +0000 ucfagls http://ucfagls.wordpress.com/?p=523 Continue reading →]]> Locally weighted scatterplot smoothing (LOWESS) or local regression (LOESS) is widely used to highlight “signal” in variables from stratigraphic sequences. It is a user-friendly way of fitting a local model that derives its form from the data themselves rather than having to be specified a priori by the user. There are generally two things that a user has to specify when using LOESS; the span or bandwidth of the local window and the degree of polynomial used in the local regression. Both control the smoothness of the fitted model, with smaller spans and higher degree polynomials giving less-smooth (more-rough) models. Usually it is just the span value that is changed, for expedience.

What can I have possibly against this? What is wrong with using LOESS for palaeo data?

The problem as I see it stems from the way palaeolimnologists choose the LOESS parameters when fitting stratigraphic data. Quite often the default is chosen in whatever software is used. Some people will play around with the span testing out some values until they get a fit that they are happy with. The more statistically savvy palaeoecologist might use a cross-validation (CV) to choose the value of span that provides the best out-of-sample predictions of the observed data. For the latter, generalised cross-validation (GCV) would normally be applied to avoid repeated fitting to each CV fold or subset.

Using the default or the value that gives you a fit that appeals to you is simply not justifiable science. The default may be totally inappropriate for the data to hand and the signal one is expecting. Furthermore, the human brain is great at seeing pattern even where non exists. The smoothness of the fitted LOESS model needs to be chosen to avoid overfitting the observed data. You can’t do that by eye!

CV or GCV should help avoid overfitting the data but importantly can only do this for independent observations in their normal incarnations. Palaeo data are far from independent observations. I’ve blogged before about the problems of smoothing temporally correlated data. Those problems apply just as well to LOESS though they are harder to solve.

Why is this an issue? Well, the reason for fitting LOESS (or any smoother/model) to the stratigraphic data is to show any pattern or trend. With LOESS, what pattern or trend you get is determined by the data and crucially by the parameters chosen for the fit. Once you have the LOESS fit you are going to look at it, ponder what it means, interpret it in light of some other factors, posit a plausible mechanism for its generation. But what if the pattern or trend you’ve lovingly produced isn’t real? What if the features being pondered are statistically indistinguishable from no trend or pattern? LOESS makes it really easy to extract a pattern or trend and because it is a proper stats method it is often taken for granted that the pattern so derived is meaningful.

Consider the example data from the earlier blog post from Kohn et al (2000), where the aim is to uncover the known model from observations drawn from this model with moderate AR(1) noise. The data sample is generated using the following code:

set.seed(321)
n <- 100
time <- 1:n
xt <- time/n
Y <- (1280 * xt^4) * (1- xt)^4
y <- as.numeric(Y + arima.sim(list(ar = 0.3713), n = n))

Several R functions can fit LOESS-like models (e.g. lowess() and loess()in base R but note these two are not the same type of LOESS model). The code chunk and figure below show three fits using different values for the span parameter.

## fit LOESS models
lo1 <- loess(y ~ xt) ## span = 0.75
lo2 <- update(lo1, span = 0.25)
lo3 <- update(lo1, span = 0.5)



Three LOESS fits to the example data using span = 0.75, 0.25 and 0.5

The plot was produced using

COL <- "darkorange1"
layout(matrix(1:4, nrow = 2))
plot(y ~ xt, xlab = expression(x[t]), ylab = expression(y[t]),
     main = expression(lambda == 0.75))
lines(Y ~ xt, lty = "dashed", lwd = 1)
lines(fitted(lo1) ~ xt, col = COL, lwd = 2)
plot(y ~ xt, xlab = expression(x[t]), ylab = expression(y[t]),
     main = expression(lambda == 0.25))
lines(Y ~ xt, lty = "dashed", lwd = 1)
lines(fitted(lo2) ~ xt, col = COL, lwd = 2)
plot(y ~ xt, xlab = expression(x[t]), ylab = expression(y[t]),
     main = expression(lambda == 0.5))
lines(Y ~ xt, lty = "dashed", lwd = 1)
lines(fitted(lo3) ~ xt, col = COL, lwd = 2)
layout(1)

There is little to choose between = 0.75 and = 0.5 if we are comparing them with the known model (the dashed line), but suppose we don’t know the true model?

The optimal according to GCV can be determined using a function modified from a posting to R-Help by Michael Friendly (the original function did more than compute GCV)

loessGCV <- function (x) {
    ## Modified from code by Michael Friendly
    ## http://tolstoy.newcastle.edu.au/R/help/05/11/15899.html
    if (!(inherits(x,"loess"))) stop("Error: argument must be a loess object")
    ## extract values from loess object
    span <- x$pars$span
    n <- x$n
    traceL <- x$trace.hat
    sigma2 <- sum(resid(x)^2) / (n-1)
    gcv  <- n*sigma2 / (n-traceL)^2
    result <- list(span=span, gcv=gcv)
    result
}

The optimize() function can be used to find the value of that achieves minimal GCV. A small wrapper function is required to link optimize() with loessGCV()

bestLoess <- function(model, spans = c(.05, .95)) {
    f <- function(span) {
        mod <- update(model, span = span)
        loessGCV(mod)[["gcv"]]
    }
    result <- optimize(f, spans)
    result
}

The optimal is chosen using bestLoess(). The optimal is about 0.18 with a GCV of around 0.009

> best
$minimum
[1] 0.1813552

$objective
[1] 0.009433405

Our original LOESS model can be updated to use this

lo.gcv <- update(lo1, span = best$minimum)

The fit this gives is shown in the figure below



Optimal LOESS fit as determined by GCV

This model clearly overfits; the result of the GCV criterion not knowing that the data are temporally autocorrelated. The whole process assumes that the data are independent and clearly palaeo data often violate this critical assumption. If you didn’t know the real underlying model the average palaeo type would already be penning their next paper on remarkable variation in [INSERT TIME PERIOD] climate from [INSERT SITE]. Yet all that wiggliness, the signal, just isn’t real. Take another sample of data and you would get about the same level of wiggliness but in different places; the signal is just a figment of the sample of data you happen to have collected.

There are solutions to this problem of course; h-block CV has been suggested as a more appropriate means of CV for time series where h observations either side of the target observation are left out from the data used to fit the model to predict the target. There are variations on approach too, as h-block CV tends to over-fit in some situations. I’ll go into this in a bit more detail in a later posting.

Be very careful using LOESS for palaeo data!

References

Kohn R., Schimek M.G., Smith M. (2000) Spline and kernel regression for dependent data. In Schimekk M.G. (Ed) (2000) Smoothing and Regression: approaches, computation and application. John Wiley & Sons, Inc.


]]> http://ucfagls.wordpress.com/2012/07/24/whats-wrong-with-loess-for-palaeo-data/feed/ 4 ucfagls loess_span_examples loess_gcv_fit http://ucfagls.wordpress.com/2012/07/17/sometimes-i-am-lost-for-words/ http://ucfagls.wordpress.com/2012/07/17/sometimes-i-am-lost-for-words/#comments Tue, 17 Jul 2012 13:00:28 +0000 ucfagls http://ucfagls.wordpress.com/?p=493 Continue reading →]]> Following on from yesterday’s momentous announcements from UK Government and RCUK on opening up access to research outputs funded by the British tax payer, John Wiley & Sons Inc. have announced the Geoscience Data Journal in partnership with the Royal Meteorological Society, the Met Office and NERC.

I first heard about this exciting new initiative via a submitted abstract from the EGU conference this summer. There need to be incentives for scientists to do more than just deposit data in repositories and data papers are a way to document their collection and processing and to acquire citations that indicate to the bean counters that scientists are making worthwhile contributions; having impact in the common parlance.

So I should be happy, right? Nope, not one bit!

Recall that RCUK and UK Government have mandated that no more restrictions than those covered by the Creative Commons By Attribution (CC BY) licence may be imposed on tax-payer funded research published in the scientific literature. CC BY is also the industry standard amongst scientists being compatible with the Budapest Open Access Initiative (BOAI) definition of open access. Guess what licence Wiley have chosen for the Geoscience Data Journal? CC BY-NC, the non-commercial derivative of CC BY.

And this is where words start to fail me…

NERC is a major supporter of the Geoscience Data Journal. It is also part of RCUK and so bound by the recent announcements on open access. UK scientists in receipt of NERC funds will no longer be allowed to publish outputs from that research in the very journal NERC is supporting and was involved in setting up. This is staggeringly short-sighted of NERC, the editorial board and Wiley. It is not as if RCUK’s decision has come out of the blue; a draft has been available for many months for consultation ( I even blogged about it a while ago).

Quite why Wiley want to retain commercial rights to papers published in the journal is beyond me. They are obviously not satisfied with pocketing the £1000 ($1500) they will charge authors to publish papers in the journal.

This should have been a PR win for Wiley and a win for the whole geosciences community. Instead it paints a picture of a publisher out of touch and out of step with the open access landscape within which it is operating.

I emailed Wiley’s Science Press Room and Rob Allan, the journal’s editor, making these points and requesting that the licence be reconsidered. If you want to do the same, email Sciencenewsroom@Wiley.com FOA of Ben Norman and Rob Allan (rob.allan AT metoffice.gov.uk). My email is appended below. I do hope that they listen…

Dear Ben, Rob,

I was alerted via Twitter to the press release from Wiley regarding the
new Geoscience Data Journal. I was excited to learn of the new journal
from an EGU abstract earlier in the year and have been looking forward
to further announcements. What should be a positive addition to the open
access landscape for the geoscience community now comes across as badly
out-dated and suggests that Wiley and the people and organisations
related to this announcement (NERC, Royal Meteorological Society etc)
are way behind the times when it comes to open access.

Why? The Geoscience Data Journal is licensed under a CC BY-NC licence.
Firstly this is not a true open access licence according to the Budapest
Open Access Initiative (BOAI) nor is it compatible with the new policy
on open access announced by Research Councils UK (of which NERC is a
part) yesterday. The BOAI has adopted CC BY as the level of restriction
that should be placed on open access research "publications". This
standard has been upheld by RCUK and the UK Government as the standard
to which it will hold recipients of its research funds. It is the height
of irony that, as it stands, no one in receipt of NERC funding would be
able to publish those funded outputs in this new journal, which NERC has
been instrumental in setting up. Similar restrictions related to
publishing EU-funded work/data are being discussed ahead of the next
major funding programme.

Springer, one of your major rivals, has adopted CC BY as the licence for
all it's open access journals. Elsevier has been publicly lambasted by
the open access movement for it's adoption of non-BOAI-compliant "open"
licences, which is but one reason that so many academics have pledged to
boycott that publisher.

By adopting the CC BY-NC licence for this journal you are severely
restricting the potential pool of authors, undermining the commonly-held
standard of CC BY for open access publishing, and going against the
ground swell of recent announcements from the UK Government and other
researcher funding agencies.

I do hope that Wiley will take another look at the licence under which
submissions to Geoscience Data Journal, and all it's open access
offerings, are made available with a view to immediately adopting CC BY
instead.

I wish you well with this exciting new initiative, but unless the
licence is reconsidered, I and many other scientists will be unable,
morally or otherwise, to submit our data to the new journal.

Yours,Dr. Gavin Simpson


]]> http://ucfagls.wordpress.com/2012/07/17/sometimes-i-am-lost-for-words/feed/ 4 ucfagls http://ucfagls.wordpress.com/2012/04/23/quantitative-palaeolimnology-my-book-chapters-are-finally-out/ http://ucfagls.wordpress.com/2012/04/23/quantitative-palaeolimnology-my-book-chapters-are-finally-out/#comments Mon, 23 Apr 2012 13:33:16 +0000 ucfagls http://ucfagls.wordpress.com/?p=461 Continue reading →]]> Today I received confirmation that the delayed fifth volume in the Developments in Palaeoenvironmental Research series has been published. The book is titled Data Handling and Numerical methods, though it covers more of the latter and, IMHO, is far more interesting than the dry title would suggest (who gets excited by Data Handling? Well, one or two people perhaps ;-)

A full table of contents can be found on the SpringerLink website, though in their infinite wisdom, this material is not available as HTML but as embedded previews of the pages, which you can download as PDFs (as you can each chapter but for the proper fee).

I authored or co-authored three of the 21 chapters

• Chapter 9 Statistical Learning in Palaeolimnology (with John Birks)
• Chapter 15 Analogue Methods in Palaeolimnology
• Chapter 19 Human Impacts: Applications of Numerical Methods to Evaluate Surface-Water Acidification and Eutrophication (with Roland Hall)


Pruned classification tree fitted to the three-fuel spheroidal carbonaceous particle (SCP) example data. The predicted fuel types for each terminal node are shown, as are the split variables and thresholds that define the prediction rules.

All three chapters relied heavily upon R; the first two being conducted entirely in R. Chapter 19 was written such a long time ago now that it wasn’t all done in R (WA-PLS, Maximum Likelihood transfer functions methods weren’t then available in R, nor were some of the ordination based methods I used). However, I’m confident the entire thing (at least the acidification parts) could be done using R now.

I have scripts for all the analyses performed using R. Some need a little work before I post them (mainly for Chapter 9) but I aim to maintain up-to-date scripts on my blog. Details soon.

Although writing these chapters took on a life of their own and used up far more time than they should have done, I am genuinely pleased with the results. I certainly learned a huge amount more about the statistics that underlay the techniques that palaeolimnologists and palaeoecologists use day in day out.

To pre-empt requests for PDFs of the chapters; I don’t have any so don’t ask. I’m not sure what arrangements were made originally with the publisher in that regard. I haven’t even seen the final book yet either; still waiting on my copy to pop through the letter box. If you want a copy and didn’t get in a pre-order at the discount rate, I suspect the best bet would be to pick on up later this summer at the International Paleolimnology Symposium in Glasgow, where I’m sure there will be a conference discount. If I hear of any offers in the meantime I’ll post something here.


]]> http://ucfagls.wordpress.com/2012/04/23/quantitative-palaeolimnology-my-book-chapters-are-finally-out/feed/ 0 ucfagls classification_tree_scp_example http://ucfagls.wordpress.com/2012/04/11/customising-vegans-ordination-plots/ http://ucfagls.wordpress.com/2012/04/11/customising-vegans-ordination-plots/#comments Wed, 11 Apr 2012 13:15:48 +0000 ucfagls http://ucfagls.wordpress.com/?p=437 Continue reading →]]> As a developer on the vegan package for R, one of the most FAQs is how to customise ordination diagrams, usually to colour the sample points according to an external grouping variable. Now, just because we get asked how to do this a lot is not really a reflection on the quality of the plot() methods available in vegan.

Ordination diagrams are difficult beasts to handle with plotting code without having an unwieldy number of arguments etc. There are potentially five sets of scores that need to be plotted so the number of arguments could quickly get out of hand if we allowed the user to pass all the relevant graphical parameters to the various sets of scores. We’ve provided a basic plot() method that is set up with useful defaults that works for all the ordination methods in vegan. This method is really there to allow the quick visualisation of the fitted ordination; I use vegan on almost a daily basis and none of my presentation or publication figures use the default plot method at all. However, we have provided all the tools needed to draw custom ordination plots within vegan. How you use them also provides a useful guide to building up base graphics plots from lower-level plotting functions. In this post I intend to show two examples of building up a simple PCA biplot from the basic building blocks available in vegan and R’s base graphics.

To get going, start R and load the vegan package. For this example I will use the classic Dutch Dune Meadow data set which I also load. A simple PCA of the species data is then fitted and stored in mod. To finish the basic example, I use the basic plot() method to plot the PCA biplot. Note that I’m using symmetric scaling here; I tend to prefer this scaling for general diagrams as it preserves many of the features of biplots without focusing on one or other sets of scores. (Here I’m also using it to illustrate how to select a scaling when you are building the plot from scratch.)

## load vegan
require("vegan")

## load the Dune data
data(dune, dune.env)

## PCA of the Dune data
mod <- rda(dune, scale = TRUE)

## plot the PCA
plot(mod, scaling = 3)

The resulting PCA biplot is shown below



The result of a call to the plot.cca() method results in a simple biplot of the Dune Meadows PCA but is not very customisable.

Building a biplot using vegan methods

The first example of a customised biplot I will show uses low-level plotting methods provided by vegan. These include points() and text() methods for objects of class "cca". (The "cca" object is one of the base ordination classes in vegan; its name is a bit unfortunate as it is the base representation for PCA, CA, RDA etc… objects — read more about this object via ?cca.object.) I am going to plot the basic PCA biplot, but colour the sites according to the land-use variable dune.env$Use, which has three levels

> with(dune.env, levels(Use))
[1] "Hayfield" "Haypastu" "Pasture"

Start by defining an object holding the desired ordination scaling and a vector of colours with which to do the plotting

scl <- 3 ## scaling = 3
colvec <- c("red2", "green4", "mediumblue")

The basic plot() method can be coerced into setting up the basic plot axes, limits etc for us by using type = "n", so we use that short cut and pass along our desired scaling

plot(mod, type = "n", scaling = scl)

The next step is to add the site scores. Here I use the points() method rather than draw the site scores using their sample ID (as is the default in the plot() method).

with(dune.env, points(mod, display = "sites", col = colvec[Use],
                      scaling = scl, pch = 21, bg = colvec[Use]))

The key point to note in the code chunk above is how I colour each site according to its land-use. I index into the vector of colours created earlier using the factor Use. Use is essentially a vector of 1s, 2s, and 3s (there are three levels remember). The colvec[Use] evaluates to a vector the same length as the number of sites, where each element is one of the pre-specified colours

 > head(with(dune.env, colvec[Use]))
[1] "green4"     "green4"     "green4"     "mediumblue"
[5] "green4"     "green4"

The display argument selects the type of scores to plot. The remainder of the arguments are the scaling for the scores (so they match the base plot) and arguments to style the plotted points.

Next I add the species scores, but this time I want to label them with (abbreviated) species names. For this I use the text() method with argument display = "species"

text(mod, display = "species", scaling = scl, cex = 0.8, col = "darkcyan")

To finish the plot I add a legend. It is important to get the ordering and labelling of the colours correct here. When I drew the site scores I used the Use factor to index the vector of plotting colours. To ensure we get the correct ordering in the legend, it is best to extract the levels as the lables, which is what I do below

with(dune.env, legend("topright", legend = levels(Use), bty = "n",
                      col = colvec, pch = 21, pt.bg = colvec))

If you want to provide custom labels, look at the levels of the factor and provide them to legend() in the correct order.

The biplot should now look like the one below



A customised ordination built from vegan lower-level plot methods

Building a biplot using base graphics directly

If you want the ultimate level of control over the plots then you will want to build the plot up from scratch using lower-level plotting functions provided by base graphics. In this second example I’ll recreate the plot I made above but from the ground up using basic plotting functions.

First, start by extracting the scores needed from the ordination object. Here the scaling required for the plot needs to be provided and the sets of scores specified

scrs <- scores(mod, display = c("sites", "species"), scaling = scl)

This results in a list with two components; sites and species

str(scrs, max = 1)

Each component is a matrix with two columns containing the scores on the first and second principal components respectively. If you want to extract scores on different axes provide the choices argument to the scores() function with a numeric vector of the axes you wish to extract. Do note that the code below assumes only two axes are extracted.

Next compute the axis limits, which need to cover the range of the site and species scores on the first (x-axis) and second (y-axis) principal components

xlim <- with(scrs, range(species[,1], sites[,1]))
ylim <- with(scrs, range(species[,2], sites[,2]))

Now everything is ready to do some actual plotting. Start preparing the plot device be starting a new plot with plot.new() and then set up the coordination system via a call to plot.window() supplying the axis limits created above. A crucial aspect of the call to plot.window() is the graphical parameter asp which controls the aspect ratio of the plot. Here we set the aspect ratio equal to 1 to preserve the relationships between scores on the different axes and the distance interpretation of the biplot

plot.new()
plot.window(xlim = xlim, ylim = ylim, asp = 1)

The reference guides (dotted lines going through the point (0,0) are added first so as to not lie on top of any of the site or species scores. Two calls to the abline() function are used

abline(h = 0, lty = "dotted")
abline(v = 0, lty = "dotted")

The next two lines of code use the default methods for points() and text() rather than the "cca" methods used above

with(dune.env, points(scrs$sites, col = colvec[Use],
                      pch = 21, bg = colvec[Use]))
with(scrs, text(species, labels = rownames(species),
                col = "darkcyan", cex = 0.8))

The legend is added in same way as before

with(dune.env, legend("topright", legend = levels(Use), bty = "n",
                      col = colvec, pch = 21, pt.bg = colvec))

All that remains is to add the plot furniture; the axes, axis labels and the plot frame

axis(side = 1)
axis(side = 2)
title(xlab = "PC 1", ylab = "PC 2")
box()

The fruits of our labours are shown below




The customised ordination built directly from base graphics elements

I don’t claim these are perfect; many of the labels lie on top of one another for example. Vegan has some functions to help with this but I’ll leave exemplifying those for another post.

The full code for the two examples is shown below

## load vegan
require("vegan")

## load the Dune data
data(dune, dune.env)

## PCA of the Dune data
mod <- rda(dune, scale = TRUE)

## plot the PCA
plot(mod, scaling = 3)

## build the plot up via vegan methods
scl <- 3 ## scaling == 3
colvec <- c("red2", "green4", "mediumblue")
plot(mod, type = "n", scaling = scl)
with(dune.env, points(mod, display = "sites", col = colvec[Use],
                      scaling = scl, pch = 21, bg = colvec[Use]))
text(mod, display = "species", scaling = scl, cex = 0.8, col = "darkcyan")
with(dune.env, legend("topright", legend = levels(Use), bty = "n",
                      col = colvec, pch = 21, pt.bg = colvec))

## or via base graphics methods
scrs <- scores(mod, display = c("sites", "species"), scaling = scl)
str(scrs, max = 1)

xlim <- with(scrs, range(species[,1], sites[,1]))
ylim <- with(scrs, range(species[,2], sites[,2]))

plot.new()
plot.window(xlim = xlim, ylim = ylim, asp = 1)
abline(h = 0, lty = "dotted")
abline(v = 0, lty = "dotted")
with(dune.env, points(scrs$sites, col = colvec[Use],
                      pch = 21, bg = colvec[Use]))
with(scrs, text(species, labels = rownames(species),
                col = "darkcyan", cex = 0.8))
axis(1)
axis(2)
title(xlab = "PC 1", ylab = "PC 2")
with(dune.env, legend("topright", legend = levels(Use), bty = "n",
                      col = colvec, pch = 21, pt.bg = colvec))
box()


]]> http://ucfagls.wordpress.com/2012/04/11/customising-vegans-ordination-plots/feed/ 45 ucfagls custom_ordination_basic_plot_method custom_ordination_vegan_version custom_ordination_base_graphics_version