Feature request: allow access to the trained scales from the layout object

[paleolimbot]

This is a fairly easy (I think) feature to add that would help me implement lazy reading of huge raster files (a stars proxy object), such that they can be downsampled appropriately. To do this, I would need access to the trained scales (for the fill and alpha aesthetics at draw time. Right now there is no way to access the trained scales except limited information about them in the panel_params and coordinates objects. I think that a reference to the ScalesList object could be included as an element of the layout object without any consequences? The Geom has access to the layout at draw time.

It is a completely valid point that this would be pushing ggplot2 where it perhaps should not go, coercing it to do things that it was not meant to do. I would argue that it is necessary if spatial rasters are ever going to be plotted in ggplot by those of us who don't have an intimate knowledge of downsampling (and maybe they shouldn't be). There is a brief discussion on the implementation of this at paleolimbot/ggspatial#17 . A reprex is included below.

library(ggplot2)
library(grid)

# in real life, this would be a reference to a large (scary) raster file
big_scary_raster <- tibble::tibble(raster = list(matrix(1:9, nrow = 3)))

StatMatrixList <- ggproto(
  "StatMatrixList",
  Stat,

  required_aes = "raster",
  default_aes = ggplot2::aes(fill = stat(z)),

  compute_layer = function(self, data, params, layout) {
    data$raster <- lapply(data$raster, function(x) {
      df <- reshape2::melt(x)
      names(df) <- c("x", "y", "z")
      df
    })

    tidyr::unnest(data, .data$raster)
  }
)

ggplot(big_scary_raster, aes(raster = raster)) +
  geom_raster(stat = StatMatrixList, hjust = 0, vjust = 0)

StatLazyMatrixList <- ggproto(
  "StatLazyMatrixList",
  StatMatrixList,

  compute_layer = function(self, data, params, layout) {
    # only return limits in the stat (these are usually cached in the raster file,
    # so the raster doesn't need to be loaded). Scales get trained based on the
    # result of this function.
    data$limits <- lapply(data$raster, function(raster) {
      tibble::tibble(
        x = c(0, ncol(raster)),
        y = c(0, nrow(raster)),
        z = range(raster)
      )
    })

    tidyr::unnest(data, .data$limits, .drop = FALSE)
  }
)

GeomLazyRaster <- ggproto(
  "GeomLazyRaster",
  Geom,
  required_aesthetics = "raster",

  default_aes = ggplot2::aes(alpha = "__default_alpha__", fill = "__default_fill__"),

  handle_na = function(data, params) {
    data
  },

  draw_panel = function(data, panel_params, coordinates) {
    # this is a super crazy hack
    # but there is no other way to get scale objects from the draw function at build time (?)
    scales <- NULL
    for(i in 1:20) {
      env <- parent.frame(i)
      if("plot" %in% names(env) && "scales" %in% names(env$plot) && inherits(env$plot$scales, "ScalesList")) {
        scales <- env$plot$scales
        break
      }
    }
    if(is.null(scales)) stop("@paleolimbot's hack to get the ScalesList from Geom$draw_panel() has failed :'(")
    fill_scale <- scales$get_scales("fill")
    alpha_scale <- scales$get_scales("alpha")

    if(all(data$alpha == "__default_alpha__")) {
      # default
      alpha <- function(x) 1
    } else if(length(unique(data$alpha)) == 1) {
      # set (or mapped but constant)
      alpha <- function(x) unique(data$alpha)
    } else if(!is.null(alpha_scale)) {
      # mapped
      alpha <- alpha_scale$map
    } else {
      stop("Don't know how to compute 'alpha'")
    }

    if(all(data$fill == "__default_fill__")) {
      # default
      fill <- function(x) 1
    } else if(length(unique(data$fill)) == 1) {
      # set (or mapped but constant)
      fill <- function(x) unique(data$fill)
    } else if(!is.null(fill_scale)) {
      # mapped
      fill <- fill_scale$map
    } else {
      stop("Don't know how to compute 'fill'")
    }

    gTree(
      raster = data$raster[[1]],
      fill = fill,
      alpha = alpha,
      coordinates = coordinates,
      panel_params = panel_params,
      cl = "lazy_raster_grob"
    )
  }
)


geom_lazy_raster <- function(mapping = NULL, data = NULL, stat = StatLazyMatrixList,
                             ..., inherit.aes = TRUE, show.legend = NA) {
  layer(
    geom = GeomLazyRaster,
    stat = stat,
    data = data,
    mapping = mapping,
    position = "identity",
    params = list(...),
    inherit.aes = inherit.aes,
    show.legend = show.legend
  )
}

makeContext.lazy_raster_grob <- function(x) {
  # here it's possible to determine height and width in inches
  # getting DPI from the graphics device may not be possible,
  # but can always fall back on a user-specified minimum

  # projection + resampling would happen here

  # apply the aesthetics
  colors <- x$fill(x$raster)
  alpha <- x$alpha(x$raster)
  colors <- paste0(colors, as.character.hexmode(scales::rescale(alpha, from = c(0, 1), to = c(0, 255))))
  dim(colors) <- dim(x$raster)

  # map the coordinates
  corners <- data.frame(x = c(0, ncol(x$raster)), y = c(0, nrow(x$raster)))
  corners_trans <- x$coordinates$transform(corners, x$panel_params)
  x_rng <- range(corners_trans$x, na.rm = TRUE)
  y_rng <- range(corners_trans$y, na.rm = TRUE)

  setChildren(x, gList(rasterGrob(
    # there is an axis irregularity between what we think of as rows
    # and what grid thinks of as rows
    aperm(colors, c(2, 1))[nrow(colors):1,],
    x = x_rng[1], y = y_rng[1],
    height = diff(y_rng), width = diff(x_rng),
    default.units = "native",
    interpolate = FALSE,
    hjust = 0,
    vjust = 0
  )))
}

ggplot(big_scary_raster, aes(raster = raster)) +
  geom_lazy_raster()

^{Created on 2019-02-03 by the reprex package (v0.2.1)}

[clauswilke]

I think this feature would also help with generating filled contours at draw time from the new binning scale @thomasp85 is implementing. See #3044 and #3096.

[thomasp85]

There are definitely enough meaningful examples to make us consider how to support it. I don’t personally think we should make the ScalesList object available as it is not currently an exported class we have committed to keeping stable. Making it available directly would limit our possibilities in the future all for the sake of a few (albeit important) needs...

[thomasp85]

An approach I would like would be some sort of method in the geom for acquiring specific scales - eg self$get_scale("fill")

[paleolimbot]

How about as a method of the layout (i.e. layout$get_scale("fill"))? I don't think the geoms ever carry any stateful info, state is passed through in the layout or panel_params. I could work up a PR implementing layout$get_scale("fill") because I think it would be minor...I would be happy to call Geom$get_scale() but I don't know how to make it happen.

[clauswilke]

I think whatever solution we come up with requires careful thought so we don't box ourselves in for future development. We've talked a lot about layer-specific scales (e.g., different color scales for different layers), and if at all possible the approach taken now should be future-proof for that. (Even if we don't really know yet how to set up layer-specific scales.)

This reasoning would suggest that it would be best if the scales could be made available through the Layer object, or directly through the Geom. I'm not immediately sure how to do this, but I think pondering a little longer is the right strategy.

[paleolimbot]

That's a great point...thanks to you both for considering this and let me know if there is anything I can do!

[clauswilke]

If you want to dig around some more in the ggplot2 code to develop potential strategies of how we can get scales info into the geom that would be helpful. Ideally, before we start coding, we'd have a list of potential approaches, each with pros and cons, and then we can make an informed decision about how to proceed.

[paleolimbot]

👍 will do!

[paleolimbot]

I tried a few things! I'm sure there are other ways to go about this...

• Make the plot/Layer/Geom stack "stateful" (i.e., instances contain backreferences to the parent object, and the Layer additionally contains a reference to the Layout). This allows Layer$get_scale(which, panel), or Geom$layer$get_scale(which, panel). Of course, this opens up a lot of the internals to possible abuse by subclasses because they would have access to the whole object through the backreferences. I've implemented a toy version of this here: master...paleolimbot:stateful-layer

library(ggplot2)

GeomScaleInfo <- ggproto(
  "GeomScaleInfo", Geom,
  required_aes = "x",
  draw_panel = function(self, data, panel_params, coord) {
    
    x_scale <- self$layer$get_scale("x", 1L)
    col_scale <- self$layer$get_scale("colour")
    
    x_range <- x_scale$range$range
    col_limits <- col_scale$range$range
    
    if(is.null(x_range)) {
      text <- "NULL x-range"
    } else {
      text <- sprintf(
        "X-scale: %s-%s\ncol-scale: %s items", 
        x_range[1], x_range[2], length(col_limits)
      )
    }
    
    grid::textGrob(text, x = 1, y = 1, hjust = 1.1, vjust = 1.5)
  }
)

geom_scale_info <- function() {
  layer(
    geom = GeomScaleInfo, stat = "identity", data = data.frame(x = 1), mapping = aes(x = x), 
    position = "identity",
    params = list(), inherit.aes = FALSE, show.legend = NA
  )
}

ggplot(mpg, aes(displ, hwy, col = class)) + geom_point() + geom_scale_info()

^{Created on 2019-02-13 by the reprex package (v0.2.1)}

• Make the Layout object have access to the non-position scales, in addition to the position scales (currently it contains the only valid reference to the trained position scales, but the non-position scales are in the original ScalesList, which also contains some confusing references to the untrained original position scales). This is more in the spirit of the current implementation, but definitely precludes anything that requires a reference to the layer (because even the Geom doesn't know anything about the layer in the current code). A toy version of that is here: master...paleolimbot:smarter-layout

library(ggplot2)

GeomScaleInfo <- ggproto(
  "GeomScaleInfo", Geom,
  required_aes = "x",
  
  # this approach requires overriding draw_layer() to pass on the layout object
  draw_layer = function(self, data, params, layout, coord) {
    if (ggplot2:::empty(data)) {
      n <- if (is.factor(data$PANEL)) nlevels(data$PANEL) else 1L
      return(rep(list(zeroGrob()), n))
    }
    
    # Trim off extra parameters
    params <- params[intersect(names(params), self$parameters())]
    
    args <- c(list(quote(data), quote(panel_params), quote(coord), quote(layout)), params)
    lapply(split(data, data$PANEL), function(data) {
      if (empty(data)) return(zeroGrob())
      
      panel_params <- layout$panel_params[[data$PANEL[1]]]
      do.call(self$draw_panel, args)
    })
  },
  
  draw_panel = function(self, data, panel_params, coord, layout) {
    
    x_scale <- layout$get_scale("x", 1L)
    col_scale <- layout$get_scale("colour")
    
    x_range <- x_scale$range$range
    col_limits <- col_scale$range$range
    
    if(is.null(x_range)) {
      text <- "NULL x-range"
    } else {
      text <- sprintf(
        "X-scale: %s-%s\ncol-scale: %s items", 
        x_range[1], x_range[2], length(col_limits)
      )
    }
    
    grid::textGrob(text, x = 1, y = 1, hjust = 1.1, vjust = 1.5)
  }
)

geom_scale_info <- function() {
  layer(
    geom = GeomScaleInfo, stat = "identity", data = data.frame(x = 1), mapping = aes(x = x), 
    position = "identity",
    params = list(), inherit.aes = FALSE, show.legend = NA
  )
}

ggplot(mpg, aes(displ, hwy, col = class)) + geom_point() + geom_scale_info()

^{Created on 2019-02-13 by the reprex package (v0.2.1)}

[clauswilke]

The first option seems nicer and more general to me. Not sure if there are any major downsides we're overlooking, though. Usually, a main design principle of software engineering is to avoid circular dependencies, and it seems this introduces one, since layer knows of geom and geom will know of layer after the change.

[thomasp85]

A general rule for most ggproto objects is that they should be stateless factories, so I would prefer a setup that keeps this

[paleolimbot]

Layer info could be included in the data (similar to how the data contains a PANEL column, it could contain a LAYER column). Then one could get a layer-specific scale using something like Layout$get_scale("x", data$PANEL[1], data$LAYER[1]) from the Geom draw methods. Without making the whole stack stateful, I can't think of a way to get Geom$get_scale() to work.

[clauswilke]

There's an additional complication: if we change the arguments of draw_layer() etc. then all the third-party geoms that have been released will break.

Is it possible to attach this information to either panel_params or coord? It doesn't belong into data.

[paleolimbot]

I don't think the arguments would need to be changed to implement Layout$get_scale("x", data$PANEL[1], data$LAYER[1])...the rare geometries that need to access the Layout would just need to override Geom$draw_layer() rather than Geom$draw_panel() or Geom$draw_group().

There's probably some other ways to get the layer ID into the draw method...the data seems like the least breaking way to do that, but it could also be set as a property of the Layer or Geom at build time.

[thomasp85]

“Just” overwriting draw_layer is not that simple as it needs to correctly split up and assemble everything efficiently. Pretty sure we could do this by adding a getter function to the param object we pass into the draw method

[clauswilke]

There's one other strategy for handing down information that we haven't considered yet: We can add new member functions that provide a geom or layer with whatever information it needs later, so that it can store it. This is less elegant than expanding the number of arguments in draw_layer()/draw_panel() but it has the advantage that it will work automatically with all existing extension packages.

[thomasp85]

Wouldn’t that still make the geoms stateful?

[paleolimbot]

Yes, that's how I implemented the stateful version of the Plot/Layer/Geom stack (a build_init(plot, layout) method that gets called from build_plot()). master...paleolimbot:stateful-layer

[clauswilke]

@thomasp85 Yes, you're right, that won't work.

I'm coming around to @paleolimbot's perspective that Layout may be the right place for this info. The panel_params object that is provided to draw_layer() etc. is probably misnamed, because it holds the geoms' own parameters. A better name would be geom_params, which immediately makes clear that this is not a place for the ggplot2 infrastructure to add generic items. The Layout, on the other hand, already holds all sorts of layer-specific info, such as the coordinate system, facets, position scales, etc:

ggplot2/R/layout.R

Lines 16 to 31 in 26cd107

	Layout <- ggproto("Layout", NULL,
	# The coordinate system and its parameters
	coord = NULL,
	coord_params = list(),
	# The faceting specification and its parameters
	facet = NULL,
	facet_params = list(),
	# A data frame giving the layout of the data into panels
	layout = NULL,
	# Per panel scales and params
	panel_scales_x = NULL,
	panel_scales_y = NULL,
	panel_params = NULL,

It could hold information about other scales as well.

I also think that requiring geoms to overwrite draw_layer() to make use of this is fine. @paleolimbot wrote the key points:

the rare geometries that need to access the Layout would just need to override Geom$draw_layer() rather than Geom$draw_panel() or Geom$draw_group().

Most geoms wouldn't need this info and thus wouldn't have to change in any way. Any geom that needs this info would have to reimplement draw_layer(), and that would include properly splitting things up and assembling them effectively. I think anybody who wants to write a special geom that can interact in a smart way with scales can also be expected to be able to reimplement draw_layer() correctly.

The alternative would be to add a new argument to draw_layer() / draw_panel() etc. that is layer-specific, e.g. layer_params. This would probably be the best change in terms of preparing us for future developments, but it would also break all existing extension packages.

[paleolimbot]

It could be that layer_params gets passed to the draw_layer() method, which is more rarely re-implemented than draw_panel() or draw_group().

[clauswilke]

Ah, maybe that's a workable idea. I'm still worried about the Layout solution because the layer parameters could in principle be different for different layers within the same panel, but Layout is the same for all layers.

[clauswilke]

Is there an easy way to find out how many extension packages have redefined draw_layer()?

[paleolimbot]

Nothing that causes any tests to fail...I'll run a revdepcheck as well.

[paleolimbot]

I forgot that revdepcheck for ggplot2 incapacitates my computer for a day...based on the preliminary results it seems unlikely it's common to override draw_layer() (no new errors related to changing the signature). Also, googling the signature of draw_layer vs googling the signature of draw_panel and draw_group seems to indicate that nobody has ever done it in a place that is google-indexed.

[paleolimbot]

How about passing the layer through the draw_layer() method? I can't think of anything to put in a layer_params object other than a reference to the layer and maybe the layer index.

master...paleolimbot:smarter-layout

library(ggplot2)

GeomScaleInfo <- ggproto(
  "GeomScaleInfo", Geom,
  required_aes = "x",
  
  # this approach requires overriding draw_layer() to pass on the layout and layer objects
  draw_layer = function(self, data, params, layout, coord, layer) {
    if (ggplot2:::empty(data)) {
      n <- if (is.factor(data$PANEL)) nlevels(data$PANEL) else 1L
      return(rep(list(zeroGrob()), n))
    }
    
    # Trim off extra parameters
    params <- params[intersect(names(params), self$parameters())]
    
    args <- c(list(quote(data), quote(panel_params), quote(coord), quote(layout), quote(layer)), params)
    lapply(split(data, data$PANEL), function(data) {
      if (empty(data)) return(zeroGrob())
      
      panel_params <- layout$panel_params[[data$PANEL[1]]]
      do.call(self$draw_panel, args)
    })
  },
  
  draw_panel = function(self, data, panel_params, coord, layout, layer) {
    
    x_scale <- layout$get_scale("x", data$PANEL[1], layer)
    col_scale <- layout$get_scale("colour", data$PANEL[1], layer)
    
    x_range <- x_scale$range$range
    col_limits <- col_scale$range$range

    text <- sprintf(
      "X-scale: %s-%s\ncol-scale: %s items", 
      x_range[1], x_range[2], length(col_limits)
    )
  
    grid::textGrob(text, x = 1, y = 1, hjust = 1.1, vjust = 1.5)
  }
)

geom_scale_info <- function() {
  layer(
    geom = GeomScaleInfo, stat = "identity", data = data.frame(x = 1), mapping = aes(x = x), 
    position = "identity",
    params = list(), inherit.aes = FALSE, show.legend = NA
  )
}

ggplot(mpg, aes(displ, hwy, col = class)) + geom_point() + geom_scale_info()

^{Created on 2019-02-19 by the reprex package (v0.2.1)}

[thomasp85]

I've lost track of what you want with layer? wasn't the intend to access scales within the draw_* methods, which you have handled by adding the get_scale method to layout

[paleolimbot]

I don't need the layer at all, unless the scale depends on it. @clauswilke seemed to indicate that layer-specific scales might occur in the future. All I need to implement lazy rasters is layout$get_scale("fill").

[clauswilke]

My concern was that get_scale() may not belong into the layout, since the layout doesn't know about layers. The best way we've so far come up with to implement layer-specific scales is to trick the geom into thinking it works with one aesthetic when it actually works with another. Something like layout$get_scale("fill") would circumvent that, because the layer wouldn't be able to intercept the call and know that fill here really should be fill2.

[paleolimbot]

There are good reasons for this in the raster context as well...I circumvent aesthetics/scales for RGB(A) rasters, but I am essentially mapping band1 to red, band2 to green, band3 to blue, and band4 to alpha, and those scales are not relevant to other layers.

I can also think of some uses for scales that depend on the panel. The position scales are currently implemented this way, and I could see how it may make sense for another aesthetic to have free scales as well with multiple panels, even if that's a long way off.

So, I think layout$get_scale("fill", panel, layer) is reasonable, even if the layer is ignored until layer-specific scales are figured out, and panel is ignored for non-position scales.

I could also see how working it into panel_params is reasonable (I think one of you suggested this already), as it currently contains the limits for the position scales (for each panel, for each layer), and is already passed to the draw method.

[yutannihilation]

Can we first implement the per-layer scale that belong to Layer, and then let the users to choose whether the scale is trained with the whole data or the layer data? If it is trained with the whole data, the per-layer scale would be the same as the plot scale.

[hadley]

It sounds like you've considered all the issues I've bought up, so I'm happy to return the reins back to y'all. (But feel free to ping me if you want more advice, regardless of my limit memory of this code)

[paleolimbot]

@clauswilke : Passing just a get_scale() function into draw_layer() seems thin to me, although I suppose a layer_params that only has a get_scale() function is about the same. I think that @yutannihilation is right that solving Layer-specific scales would be helpful to this problem.

In particular, it may better define the role of the Layer to the point where it could get passed to the draw_layer() method? I'm not sure this means it has to be "public"...it would be one level higher than anybody has ever overridden, plus the layout is exposed to the draw method as well, and has a similar level of definition as the layer (I think). For my use case, this would allow something like hillshades or RGB(A) to be Stats that are computed at draw time rather than Geoms that are basically a stat + a geom.

Alternatively, define the layer_params as everything that is computed by the Layer + plot + layout but not stored there (aesthetics, initial data, final data, scales) and make that a well-defined public class?

[clauswilke]

I don't think we should export Layer at this point, because it may require substantial revisions in the future. Thus, we can't hand the layer to the geom. (And also, it wouldn't be a clean design, since it creates a circular dependency.)

Adding more things to layer_params, in case they may be useful in the future, is fine by me. Some of that information could be captured in this function:

ggplot2/R/layer.r

Lines 204 to 206 in 033fb52

	setup_layer = function(self, data, plot) {
	data
	},

I'm not sure of the performance implications if every layer stores the raw data, though. Are these all just references that don't cost anything?

[clauswilke]

Actually, let me qualify my previous comment: I'm somewhat concerned about adding the aesthetic mapping to the layer_params, because at some point we may want to disentangle geom and stats mappings. At present that would be limited to changes in Layer, but if we expose this in layer_params then we're essentially locked in for all eternity. I think a more conservative approach is better. Let's only add to layer_params what we feel confident is useful and won't likely change in the future. It's always easier to add other things if the need arises than to take things back that were ill-conceived.

[paleolimbot]

Got it! Something like this?

master...paleolimbot:layer-layer-params

I think I stepped in a deep issue for my second ever in ggplot2...I have a feeling this is better suited for somebody who is more familiar with the future needs of ggplot2. I will happily use whatever it's decided is best for this (or I won't use anything if it's decided it's too much of a hack).

[clauswilke]

Does it work? I don't remember if the scales are available at that point in the plot build. In any case, I think you've mostly solved the issue. It's just details at this point.

@thomasp85 Do you have an opinion on whether the layer_params object should be a list or a ggproto object?

[yutannihilation]

I don't think we should export Layer at this point, because it may require substantial revisions in the future. Thus, we can't hand the layer to the geom. (And also, it wouldn't be a clean design, since it creates a circular dependency.)

Hmm, agreed. Even the per-layer scales were to be implemented as the one that belongs to Layer, we need some some interface/convention like layer_params between geom. Sorry for the noise....

[clauswilke]

I created a separate issue for the LayerSf() problem discussed in the hidden comments. Let's move any further discussion of that topic there: #3158

[clauswilke]

@paleolimbot Does the latest version of your code solve the problem you set out to solve? If yes, then I think you should open a PR and then we can resolve any remaining issues.

Note that the PR will require a news item, as described here: https://style.tidyverse.org/news.html#bullets

[paleolimbot]

Sorry, I lost track of the thread and had to Ph.D. for a few days. Yes, the latest version of my code solves the problem, although I haven't unit tested yet. I will add some tests and the news item and PR after I defend my comps (today)!

[clauswilke]

Good luck for your comps!

[teunbrand]

The example in #3116 (comment) works because geom_point() is training the scale, and it is just a 'reporter' geom instead of one that also needs to train/map/use the scale. The mechanism by which a scale should be trained is not very clear to me. If the data which requires delayed mapping has e.g. range 0-10, but a different layer has range 20-30 for the same aesthetic, then the scale will be trained as 20-30 and will not result in a good delayed mapping. If the scale is trained on the spot to have 0-30 range, then the second layer will be mismapped.

[thomasp85]

Closing this as it seems stale. We can reopen if anyone feels strongly about it