The model

The model has two identical firms selling differentiated products to socially-conscientious consumers with heterogeneous preferences. Firms are able to engage in socially responsible behavior (e.g., voluntarily abating harmful emissions), with the twist that their motivation for doing so can be both for the usual strategic reasons and due to the preferences of the firm’s management—a moral manager. A greater degree of moral management means that a firm places more weight on pro-social behavior relative to profit, whereas a profit-maximizing manager places no weight on pro-social behavior and only considers profit. An increase in the degree of moral management for a firm affects the amount of pro-social behavior by both firms in the market, and consequently affects prices and expenditure/revenue shares for both products.

Part of the solution for the model is a pair of price and quantity functions that give the prices and market shares as a function of the degree of morality for each firm’s manager. These can be plugged into a price index to give a measure of the change in price from a change in managers’ preferences over time. In particular, comparing prices when managers are profit maximizers to prices with moral managers give a measure of the change in price due to moral managers.

Impact of morality on price

Let’s start with the simplest case: the two firms each have the same type of manager. In this case prices will be the same for both firms’ outputs (which do not change over time), so any price index that satisfies the proportionality axiom will simply be the change in price over time. Equations (2) and (3) in the paper can be used to establish that this change in price over time is a linear function of the degree to which managers value the social impact of their firm’s behavior, , where is a complex function of the parameters of the model and is the degree to which the manager values the social conduct of the their firm (relative to profits). Therefore, an increase in the morality of managers in the market will translate into an increase in any reasonable price index over time.

The more interesting case is when firms have different types of managers. In this case, an increase in the morality of one firm’s management will lead to an increase in price and quantity for that firm’s product and a (smaller) decrease in price and quantity for the competitor’s product. The increase in price for one product and decrease in price for the other work against each other to dampen the increase in price when firms are run by the same type of manager. Note that the expenditure/revenue share increases for the product with the increasing price, the opposite of what’s usually expected, and this means that the Paasche index is larger than the Laspeyres index.

Rather than try to get a closed-form expression for a price index as a function of the degree of moral management in the market, let’s work through some numerical examples to see how it compares to the case when both managers have the same type. We’ll start by making a class to generate price relatives and expenditure shares for a pair of types, relative to profit-maximizing managers.

import pandas as pd
import numpy as np
from typing import Callable


class ConstraintError(Exception):
    pass


class MoralManagers:
    def __init__(self, t: float, v: float, c: float, k: float, F: float) -> None:
        t = float(t)
        v = float(v)
        c = float(c)
        k = float(k)
        F = float(F)
        if t <= (v - k) ** 2 / (3 * F):
            raise ConstraintError("t is too small")

        self.phi = (9 * t * F - (v - k) ** 2) / (9 * t * F - 2 * (v - k) ** 2)
        self.rho = (v - k) ** 2 / (9 * t * F - (v - k) ** 2)
        self.t = t
        self.v = v
        self.c = c
        self.k = k
        self.F = F

        base_s = (self.v - self.k) / (3 * self.F)
        self.base_p = self.p0(base_s, base_s)

    def p0(self, s0: float, s1: float) -> float:
        return (
            self.t
            + self.c
            + (self.v + 2 * self.k) / 3 * s0
            - (self.v - self.k) / 3 * s1
        )

    def p1(self, s0: float, s1: float) -> float:
        return self.p0(s1, s0)

    def q0(self, p0: float, p1: float, s0: float, s1: float) -> float:
        return 0.5 + (self.v * (s0 - s1) - p0 + p1) / (2 * self.t)

    def q1(self, p0: float, p1: float, s0: float, s1: float) -> float:
        return self.q0(p1, p0, s1, s0)

    def s0(self, mu0: float, mu1: float) -> float:
        return (self.v - self.k) / (3 * self.F) + self.phi / self.F * (
            mu0 - self.rho * mu1
        )

    def s1(self, mu0: float, mu1: float) -> float:
        return self.s0(mu1, mu0)

    def pi(self, p: float, q: float, s: float) -> float:
        return (p - self.c - self.k * s) * q - self.F / 2 * s**2

    def pqs(self, mu0: float, mu1: float) -> dict:
        s0 = self.s0(mu0, mu1)
        s1 = self.s1(mu0, mu1)
        p0 = self.p0(s0, s1)
        p1 = self.p1(s0, s1)
        q0 = self.q0(p0, p1, s0, s1)
        q1 = self.q1(p0, p1, s0, s1)
        pi0 = self.pi(p0, q0, s0)
        pi1 = self.pi(p1, q1, s1)
        share0 = p0 * q0 / (p0 * q0 + p1 * q1)
        if pi0 < 0:
            raise ConstraintError("negative profit for firm 0")
        elif pi1 < 0:
            raise ConstraintError("negative profit for firm 1")
        else:
            return {
                "rel0": p0 / self.base_p,
                "rel1": p1 / self.base_p,
                "share0": share0,
            }


def laspeyres_index(rel0: float, rel1: float, share0: float) -> float:
    return rel0 * 0.5 + rel1 * 0.5


def paasche_index(rel0: float, rel1: float, share0: float) -> float:
    return 1 / (share0 / rel0 + (1 - share0) / rel1)


def fisher_index(rel0: float, rel1: float, share0: float) -> float:
    return (laspeyres_index(rel0, rel1, share0) * paasche_index(rel0, rel1, share0))**0.5


def tornqvist_index(rel0: float, rel1: float, share0: float) -> float:
    return rel0 ** (0.5 * 0.5 + 0.5 * share0) * rel1 ** (0.5 * 0.5 + 0.5 * (1 - share0))


def make_index(data: pd.DataFrame, index: Callable) -> pd.Series:
    return data.apply(lambda x: index(x["rel0"], x["rel1"], x["share0"]), axis=1)

We can now simulate the impact of an increase in morality for one firm on a Fisher index under three scenarios: managers are entirely uncorrelated, so that the other firm is always a profit maximizes; managers are somewhat correlated; and managers are perfectly correlated and have the same type. In order to respect the constraint that firms must make positive profit, we’ll only consider a change in morality up to the point at which the manager places equal value on profitability and social conduct.

def simulate(corr: float, model: MoralManagers) -> pd.DataFrame:
    df = [
        pd.DataFrame(model.pqs(m, corr * m), index=[m]) for m in np.arange(0, 1.05, 0.05)
    ]
    return pd.concat(df)


mm = MoralManagers(3, 2, 0.25, 0.25, 1)

simulation = {corr: simulate(corr, mm) for corr in [0, 0.5, 1]}

fisher = pd.concat(
    {f"corr={corr}": make_index(s, fisher_index) for corr, s in simulation.items()},
    axis=1,
)
fisher.plot(ylabel="Fisher index", xlabel="Degree of morality")

The top line shows the linear effect of morality on price when both firms have the same type of manager. Introducing heterogeneity in the types of managers results in a smaller increase in price, although the Fisher index is now a convex function of the degree of morality, with the curvature becoming more pronounced as managers’ types become less correlated. This is driven by the Paasche index, as the Laspeyres index is a linear function of the degree of morality.

df = pd.DataFrame(
    {
        "Laspeyres index": make_index(simulation[0], laspeyres_index),
        "Paasche index": make_index(simulation[0], paasche_index),
    }
)

df.plot(ylabel="Index", xlabel="Degree of morality")

Using a Törnqvist index instead of a Fisher index gives the same general result.

tornqvist = pd.concat(
    {f"corr={corr}": make_index(s, tornqvist_index) for corr, s in simulation.items()},
    axis=1,
)
tornqvist.plot(ylabel="Törnqvist index", xlabel="Degree of morality")

I find this to be an interesting result. Despite coming from a model in which both firms and consumers are making optimal decisions, an increase in the degree of morality from one firm in the market results in price indexes that behave quite differently than expected.

Weighted time product dummy

Let’s start with the WTPD index. This index is recovered from the coefficient on a set of time-dummy variables in a weighted log-linear regression model with a dummy variable for each product (e.g., Diewert and Fox 2022, eq. 9). Fitting this model is impractical with many products—with observations for products over time periods, the design matrix is a matrix and will consume at least bytes of memory. Fortunately, only the coefficients on the time-dummy variables are required to make the index and so the product-dummy variables can be removed by treating them as fixed effects and demeaning the regression equation (Wooldridge 2002, sec. 10.5). Now the design matrix has only columns and, as is usually much smaller than , will consume far less memory.

Let’s consider a example. We’ll start with a function to make (unbalanced) transaction data for a collection of products over time.

set.seed(1552443)

make_prices <- function(products, periods, frac = 0.1) {
  res <- data.frame(
    product = as.factor(seq_len(products)),
    time = as.factor(rep(seq_len(periods), each = products)),
    price = rlnorm(products * periods),
    quantity = runif(products * periods)
  )
  res[-sample(nrow(res), frac * nrow(res)), ]
}

head(make_prices(10, 13))

  product time     price  quantity
2       2    1 1.4944618 0.2978414
3       3    1 0.6778284 0.9263743
5       5    1 1.2611687 0.3008513
7       7    1 0.7996127 0.9532272
8       8    1 0.2138277 0.4409094
9       9    1 1.9830544 0.9422246

We’ll also make a function to take these data and construct the WTPD index from a fixed-effects model with the {fixest} package.

wtpd_index <- function(prices) {
  prices <- prices |>
    dplyr::group_by(time) |>
    dplyr::mutate(w = gpindex::scale_weights(price * quantity))

  mdl <- fixest::feols(
    log(price) ~ time | product,
    data = prices,
    weights = ~w
  )

  exp(coef(mdl))
}

Now let’s build a WTPD index with data for 100,000 products over 25 time periods. Doing this with a regular linear model, where each product has its own dummy variable, would require 1.8 TB of memory to just store the design matrix, making it impractical to calculate. Using a fixed-effects model to recover the WTPD index can be done without issue on any decent laptop.

price_data <- make_prices(1e5, 25)

bench::mark(fixest = wtpd_index(price_data))

# A tibble: 1 × 6
  expression      min   median `itr/sec` mem_alloc `gc/sec`
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
1 fixest        4.17s    4.17s     0.240    2.77GB    0.960

Note that this is generally much faster than the routine for making the WTPD index in the excellent {IndexNumR} package.

wtpd_index2 <- function(prices, window) {
  # IndexNumR expects these to be integers.
  prices[c("time", "product")] <- lapply(
    prices[c("time", "product")],
    as.integer
  )
  res <- IndexNumR::WTPDIndex(
    prices,
    "price",
    "quantity",
    "time",
    "product",
    window = window
  )
  as.numeric(res[-1])
}

price_data <- make_prices(50, 25)

bench::mark(
  fixest = unname(wtpd_index(price_data)),
  IndexNumR = wtpd_index2(price_data, 25)
)

# A tibble: 2 × 6
  expression      min   median `itr/sec` mem_alloc `gc/sec`
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
1 fixest       8.77ms   9.56ms   103.       1.51MB     3.98
2 IndexNumR     3.59s    3.59s     0.278  102.15MB     8.07

Geary-Khamis index

The GK index is another multilateral index that’s often written as a function of a large matrix. Following Diewert and Fox (2022), the GK index can be written as a function of two matrices. One of these matrices is diagonal, so it can be handled efficiently, but the other is usually dense (unless product imbalanced is really bad) and is impractical to store in memory. Using the same data as with the WTPD index, it would require 80 GB of memory to just store this matrix.

An alternative approach (Balk 2008, 245–46) expresses the GK index as a function of three matrices. With the same data as before, this now requires only 15 KB of memory. Here is relatively fast function to implement this approach.

#' Intertemporal Geary-Khamis index
#'
#' Make a multilateral Geary-Khamis price index.
#'
#' @param p A numeric vector of prices.
#' @param q A numeric vector of quantities.
#' @param period A factor, or something that can be coerced into one, that
#'   gives the corresponding time period for each element in `p` and
#'   `q`. The ordering of time periods follows the levels of `period`
#'   to agree with [`cut()`][cut.Date].
#' @param product A factor, or something that can be coerced into one, that
#'   gives the corresponding product identifier for each element in `p` and
#'   `q`.
#' @param na.rm Should missing values be removed? By default they are not.
#'
#' @returns
#' A numeric vector of period-over-period price indexes.

fast_gk <- function(p, q, period, product, na.rm = FALSE) {
  period <- as.factor(period)
  product <- as.factor(product)

  qs <- unsplit(lapply(split(q, product), gpindex::scale_weights), product)
  attributes(product) <- NULL # faster to match on numeric codes
  ux <- unique(product)
  product <- lapply(
    split(product, period),
    \(x) match(ux, x, incomparables = NA)
  )

  v <- Map(`[`, split(p * q, period), product)
  qs <- Map(`[`, split(qs, period), product)

  n <- nlevels(period)
  # Make the matrices from pp. 245-246 of Balk (2008) to build the quantity
  # index, then deflate.
  m <- vector("list", n)
  for (i in seq_along(m)) {
    m[[i]] <- vapply(
      qs,
      \(qs) sum(qs * gpindex::scale_weights(v[[i]]), na.rm = na.rm),
      numeric(1L)
    )
  }
  e <- diag(n)
  r <- cbind(rep.int(1, n), matrix(0, n, n - 1))
  q <- solve(do.call(rbind, m) - e + r)[1L, ]
  v <- vapply(v, sum, numeric(1L), na.rm = na.rm)
  # Return the period-over-period index.
  v[-1L] / v[-length(v)] / (q[-1L] / q[-length(q)])
}

As with the WTPD index, any decent laptop can compute this index.

gk_index <- function(prices) {
  fast_gk(
    prices$price,
    prices$quantity,
    prices$time,
    prices$product,
    na.rm = TRUE
  )
}

price_data <- make_prices(1e5, 25)

bench::mark(fast_gk = gk_index(price_data))

# A tibble: 1 × 6
  expression      min   median `itr/sec` mem_alloc `gc/sec`
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
1 fast_gk       1.37s    1.37s     0.729     819MB    0.729

It is similarly non-trivial to make an index of this size with {IndexNumR}.

gk_index2 <- function(prices, window) {
  prices[c("time", "product")] <- lapply(
    prices[c("time", "product")],
    as.integer
  )
  res <- IndexNumR::GKIndex(
    prices,
    "price",
    "quantity",
    "time",
    "product",
    window = window
  )
  as.numeric(res[-1] / res[-length(res)])
}

price_data <- make_prices(1000, 25)

bench::mark(
  fast_gk = unname(gk_index(price_data)),
  IndexNumR = gk_index2(price_data, 25)
)

# A tibble: 2 × 6
  expression      min   median `itr/sec` mem_alloc `gc/sec`
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
1 fast_gk      7.23ms   7.57ms   125.       8.15MB    1.99 
2 IndexNumR      1.4s     1.4s     0.713  332.21MB    0.713

Repeat sales

The repeat-sales indexes listed by Shiller (1991) are a type of multilateral index usually found in the housing economics literature. Like the WTPD index, they come from linear models and suffer from the same large design matrices. The geometric repeat-sales index is like the WTPD index, although usually with a much longer time horizon, except that it uses the first-difference transformation to sweep out the product fixed effects instead of the within-group transformation. The arithmetic repeat-sales index is more complicated as it comes from an IV estimator and its not obvious, at least to me, how the matrices for this index can be made smaller.

Instead of reducing the dimensionality of the repeat-sales matrices, these matrices are naturally sparse when products sell infrequently (like housing) and can be stored as sparse matrices. ¹ Let’s adapt the vignette from my {rsmatrix} package to see this in action with some data for 5 million house sales over 30 years.

periods <- seq(as.Date("2000-01-01"), as.Date("2029-12-31"), "day")

prices <- data.frame(
  sale = sample(periods, 5e6, TRUE),
  property = factor(sprintf("%06d", sample(1:1e6, 5e6, TRUE))),
  price = rlnorm(5e6)
) |>
  dplyr::mutate(period = cut(sale, "month"))

sales_pairs <- rsmatrix::rs_pairs(prices$sale, prices$property)
prices[c("price_prev", "period_prev")] <- prices[
  sales_pairs,
  c("price", "period")
]

prices <- prices |>
  dplyr::mutate(
    holding_period = as.numeric(period) - as.numeric(period_prev)
  ) |>
  dplyr::filter(holding_period > 2)

head(prices)

        sale property     price     period price_prev period_prev
1 2016-05-11   627686 0.7705151 2016-05-01  0.1878927  2014-06-01
2 2029-07-22   992771 0.2366689 2029-07-01  0.4822262  2026-10-01
3 2011-07-17   925847 0.1712451 2011-07-01  0.5992709  2003-05-01
4 2013-03-28   337973 0.1540939 2013-03-01  0.9106525  2002-11-01
5 2008-04-22   295262 0.3274417 2008-04-01  0.2760829  2003-10-01
6 2015-12-27   038822 2.2174534 2015-12-01  1.7673652  2010-11-01
  holding_period
1             23
2             33
3             98
4            124
5             54
6             61

We can now build a generator to make the sparse repeat-sales matrices. If these were dense matrices they would consume 22.1 GB of memory. This isn’t nearly as bad as the WTPD or GK indexes, but still enough to make computing these indexes cumbersome. With sparse matrices, however, it is easy to make the repeat-sales index.

matrices <- with(
  prices,
  rsmatrix::rs_matrix(period, period_prev, price, price_prev, sparse = TRUE)
)

Z <- matrices("Z")
X <- matrices("X")
Y <- matrices("Y")

bench::mark(ars = 1 / Matrix::solve(crossprod(Z, X), crossprod(Z, Y)))

# A tibble: 1 × 6
  expression      min   median `itr/sec` mem_alloc `gc/sec`
  <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
1 ars           1.53s    1.53s     0.652       3MB        0

GEKS

Let’s conclude by briefly mentioned the GEKS family of multilateral indexes. As with the others, the GEKS depends on a matrix, but it is only a matrix and will require very little memory. This makes GEKS indexes somewhat more straightforward to compute than the other popular multilateral indexes.

R version 4.5.2 (2025-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 24.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.12.0 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Toronto
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Matrix_1.7-4        jsonlite_2.0.0      dplyr_1.1.4        
 [4] compiler_4.5.2      rsmatrix_0.2.9      tidyselect_1.2.1   
 [7] Rcpp_1.1.0          yaml_2.3.10         fastmap_1.2.0      
[10] lattice_0.22-7      R6_2.6.1            generics_0.1.4     
[13] Formula_1.2-5       knitr_1.50          htmlwidgets_1.6.4  
[16] tibble_3.3.0        pillar_1.11.1       rlang_1.1.6        
[19] utf8_1.2.6          xfun_0.54           stringmagic_1.2.0  
[22] fixest_0.13.2       cli_3.6.5           magrittr_2.0.4     
[25] digest_0.6.38       grid_4.5.2          IndexNumR_0.6.0    
[28] sandwich_3.1-1      gpindex_0.6.3       lifecycle_1.0.4    
[31] nlme_3.1-168        vctrs_0.6.5         bench_1.1.4        
[34] evaluate_1.0.5      glue_1.8.0          data.table_1.17.8  
[37] numDeriv_2016.8-1.1 zoo_1.8-14          profmem_0.7.0      
[40] dreamerr_1.5.0      rmarkdown_2.30      tools_4.5.2        
[43] pkgconfig_2.0.3     htmltools_0.5.8.1  

Turning two regressions into one

The hedonic imputation model starts with two linear models, one for transactions in period 0,

and one for transactions in period 1,

where is a vector of characteristics that confound a change in price over time with a change in the composition of housing.

The Laspeyres hedonic imputation index, , compares the predicted prices from Equation 2 against those from Equation 1 over the distribution of characteristics in period 0

where is the vector of average period 0 characteristics. This is equation 5.19 in ILO et al. (2013).

Equation 1 and Equation 2 can be nested to get a single linear model. Subtracting from the interaction term then gives Equation 3 as a coefficient on a time dummy variable

where . The Paasche hedonic imputation index follows along the same lines, replacing with . The Fisher hedonic imputation index combines these two, which is the same as replacing with . Note that this reduces to the classic time-dummy hedonic index when .¹

An example

Let’s go through an example to see this in action. As the goal is just to give an example, I’ll make up some typical-looking transaction data for property sales with a few characteristics over two periods.

set.seed(15243)

sales <- data.frame(
  period = rep(0:1, c(120, 180)),
  price = rlnorm(300),
  area = runif(300, 800, 2500),
  age = sample(1:80, 300, replace = TRUE),
  amenity_score = runif(300)
)

head(sales)

  period     price      area age amenity_score
1      0 1.8931980 2415.7592  70    0.01447159
2      0 0.9737266 2244.8349   5    0.59602963
3      0 0.9860096 1937.4438  80    0.39965375
4      0 1.4830738  989.7379  66    0.16038727
5      0 0.3651522 2324.3656  50    0.82509594
6      0 0.7955844 1930.3795  12    0.38836734

We’ll work with the following basic model relating price with characteristics in each period

The two-regression approach for making the Laspeyres-imputation index fits two separate models for each time period and compares the average ratio of predicted prices over the distribution of housing characteristics for sales in period 0.

sales2 <- split(sales, ~period)

mdl1 <- lm(log(price) ~ age + area + I(area^2) + amenity_score, sales2[[1]])

mdl2 <- lm(log(price) ~ age + area + I(area^2) + amenity_score, sales2[[2]])

laspeyres <- exp(mean(predict(mdl2, sales2[[1]]) - predict(mdl1, sales2[[1]])))

laspeyres

[1] 0.9367763

The Paasche-imputation index does the same thing, just over the distribution of characteristics for sales in period 1, and the Fisher-imputation index combines the Laspeyres and Paasche indexes.

paasche <- exp(mean(predict(mdl2, sales2[[2]]) - predict(mdl1, sales2[[2]])))

paasche

[1] 0.9376688

fisher <- sqrt(laspeyres * paasche)

fisher

[1] 0.9372224

To make these indexes with one regression we need to remove the appropriate mean from the characteristics variables and interact them with the time dummy in the linear model. In each case the coefficient on the time dummy variable gives the same index as using two regressions. For simplicity, I’ll just show the case of the Fisher index.

dm <- function(x) {
  m0 <- mean(x[eval(substitute(period == 0), parent.frame())])
  m1 <- mean(x[eval(substitute(period == 1), parent.frame())])
  x - mean(c(m0, m1))
}

mdl_fisher <- lm(
  log(price) ~ period +
    age +
    area +
    I(area^2) +
    amenity_score +
    period:(dm(age) + dm(area) + dm(area^2) + dm(amenity_score)),
  sales
)

all.equal(exp(coef(mdl_fisher)["period"]), fisher, check.attributes = FALSE)

[1] TRUE

Representing the Fisher index as the coefficient in a linear model also makes it easy to get the (delta method) standard error for the index. ²

exp(coef(mdl_fisher)["period"]) * sqrt(sandwich::vcovHC(mdl_fisher)[2, 2])

   period 
0.1009944 

Adding weights to both the regression model and the index-number formulas is a simple extension. Continuing with the above example, suppose we want to weight by house price so that more expensive houses received a larger weight.

mdl1 <- lm(
  log(price) ~ age + area + I(area^2) + amenity_score,
  sales2[[1]],
  weights = price
)

mdl2 <- lm(
  log(price) ~ age + area + I(area^2) + amenity_score,
  sales2[[2]],
  weights = price
)

laspeyres <- exp(
  weighted.mean(
    predict(mdl2, sales2[[1]]) - predict(mdl1, sales2[[1]]),
    sales2[[1]]$price
  )
)

paasche <- exp(
  weighted.mean(
    predict(mdl2, sales2[[2]]) - predict(mdl1, sales2[[2]]),
    sales2[[2]]$price
  )
)

fisher <- sqrt(laspeyres * paasche)

Making the Fisher index with one regression just requires removing the weighted mean from the characteristics variables in the interaction term.

dm <- function(x) {
  m0 <- eval(
    substitute(weighted.mean(x[period == 0], price[period == 0])),
    parent.frame()
  )
  m1 <- eval(
    substitute(weighted.mean(x[period == 1], price[period == 1])),
    parent.frame()
  )
  x - mean(c(m0, m1))
}

mdl_fisher <- lm(
  log(price) ~ period +
    age +
    area +
    I(area^2) +
    amenity_score +
    period:(dm(age) + dm(area) + dm(area^2) + dm(amenity_score)),
  sales,
  weights = price
)

all.equal(exp(coef(mdl_fisher)["period"]), fisher, check.attributes = FALSE)

[1] TRUE

Why two regressions can be better

As noted by Aizcorbe (2014, chap. 3), constructing the hedonic-imputation index with two regressions is more flexible than recovering an index from a regression coefficient. The trick of doing it with one regression works because the model is linear; using a non-linear regression model (however rare) may require fitting two models separately. Explicitly using the fitted values from two models to make price relatives also allows for the use of other index-number formulas, not just geometric ones.

One reason that I’ve not seen for the two-regression approach is that it’s also more convenient to calculate product contributions when doing two regressions.

laspeyres_relatives <- exp(
  predict(mdl2, sales2[[1]]) - predict(mdl1, sales2[[1]])
)
contrib_laspeyres <- gpindex::geometric_contributions(laspeyres_relatives)

paasche_relatives <- exp(
  predict(mdl2, sales2[[2]]) - predict(mdl1, sales2[[2]])
)
contrib_paasche <- gpindex::geometric_contributions(paasche_relatives)

contrib_fisher <- Map(
  `*`,
  gpindex::transmute_weights(0, 1)(c(laspeyres, paasche)),
  list(contrib_laspeyres, contrib_paasche)
)

plot(
  density(unlist(contrib_fisher)),
  main = "Distribution of sales contributions"
)

Setup

Let’s start with a simple two-commodity model where a representative consumer has Cobb-Douglas preferences. We’ll make a small class and mixin to make a Laspeyres and Fisher index given prices based on these preferences.

import numpy as np
import matplotlib.pyplot as plt


class IndexMixin:
    def laspeyres(self, p1: np.ndarray, p0: np.ndarray) -> float:
        q0 = self.demand(p0)
        return np.sum(p1 * q0) / np.sum(p0 * q0)

    def paasche(self, p1: np.ndarray, p0: np.ndarray) -> float:
        q1 = self.demand(p1)
        return np.sum(p1 * q1) / np.sum(p0 * q1)

    def fisher(self, p1: np.ndarray, p0: np.ndarray) -> float:
        laspeyres = self.laspeyres(p1, p0)
        paasche = self.paasche(p1, p0)
        return np.sqrt(laspeyres * paasche)


class CobbDouglas(IndexMixin):
    def __init__(self, alpha: np.ndarray, m: float):
        self._alpha = np.asarray(alpha, dtype=np.float64)
        self._alpha /= np.sum(self._alpha)
        self._m = float(m)

    def demand(self, p: np.ndarray) -> np.ndarray:
        if len(p) != len(self._alpha):
            raise ValueError(f"must supply prices for {len(self._alpha)} products")
        return self._alpha * self._m / p

We’ll also make a function to oscillate prices. The idea is to form a chain of index values that starts with both products having a price of [1, 1], switches prices between [1, 1.1] and [1.1, 1] some number of times, and ends with prices back at [1, 1]. As a chained index is just the cumulative product of these index values, we can decompose the chained index into the product of the first and last change in price, and the product of the oscillations. What’s necessary to drive inflation is that the product of the oscillations is greater than 1.

We’ll also make a second function that does a more complex oscillation that we’ll use later.

def bounce(index) -> tuple[np.float64]:
    initial = [1, 1]
    p1 = [1, 1.1]
    p2 = [1.1, 1]
    i = index(p1, initial) * index(initial, p1)
    b = index(p2, p1) * index(p1, p2)
    return i, b


def bounce2(index) -> tuple[np.float64]:
    initial = [1, 1]
    p1 = [1, 1.1]
    p2 = [1.2, 1]
    p3 = [1.1, 1]
    i = index(p1, initial) * index(initial, p1)
    b = index(p2, p1) * index(p3, p2) * index(p1, p3)
    return i, b

Inflation due to measurement

Let’s start by generating some preferences and giving our representative consumer a fixed income, then simulating how the chained Laspeyres index evolves as prices bounce around.

u = CobbDouglas([1, 2], 100)

res = bounce(u.laspeyres)

oscillate = [res[0] * res[1]**n for n in range(20)]

fig, ax = plt.subplots()
ax.plot(oscillate)
plt.xticks(range(0, 20, 2))
plt.show()

Here we see that, despite prices starting and ending at the same level, the chained Laspeyres index shows that prices have increased over time. This happens because each oscillation registers an increase in prices (about 0.8%), and these compound to show that prices are increasing over time. Note that the culprit here is the well-known substitution bias in the Laspeyres index; if there was no substitution bias then the index would be transitive with Cobb-Douglas preferences and (correctly) show no change in prices over time. Instead, this happens with Leontief preferences.

class Leontief(IndexMixin):
    def __init__(self, m: float):
        self._m = float(m)

    def demand(self, p: np.ndarray) -> np.ndarray:
        return np.repeat(self._m / np.sum(p), len(p))

bounce(Leontief(100).laspeyres)

(np.float64(1.0), np.float64(1.0))

Both components of the chained index—the first and last change in price and the part due to oscillations—show no change in price.

We can get the same behavior from the Fisher index, albeit with a more complex form of oscillation. (Using the simple oscillation would show no change in prices over time because the Fisher index satisfies the time-reversal property.)

res = bounce2(u.fisher)

oscillate = [res[0] * res[1] ** n for n in range(20)]

fig, ax = plt.subplots()
ax.plot(oscillate)
plt.xticks(range(0, 20, 2))
plt.show()

Each oscillation registers a 0.02% increase in prices. Note that this behavior from the Fisher index is sensitive to the parameters in the Cobb-Douglas utility function and it’s also possible to get that prices decreases over time.

u = CobbDouglas([2, 1], 100)
res = bounce2(u.fisher)
print(res)

(np.float64(1.0), np.float64(0.9998308882582081))

Overall, this example shows that more than just money supply can drive our measurement of inflation, even if increasing money supply is the only true cause of inflation.

What is diversity, anyway?

Diversity is a tricky concept to formalize. Up until writing this post, I would have said it was just the number of species in an ecosystem. In his seminal paper, Hill (1973) considers a family diversity indexes that give a measure of the effective number of species in an ecosystem—that is, the number of species that would be present in an ecosystem if all species were equally prevalent. Mathemetically, if there are species and the -th species appears with probability , then the diversity index of order is

Different values for yield different indexes; for example, measures diversity by the total number of species , also known as the richness of the ecosystem. As a function of , is a continuously decreasing function that maps values in onto . We can see this with an example of an ecosystem with 10 species.

diversity_index <- function(x, alpha) {
  if (alpha != 1) {
    sum(x^alpha)^(1 / (1 - alpha))
  } else {
    exp(-sum(p * log(p)))
  }
}

set.seed(54321)

p <- sort(gpindex::scale_weights(rlnorm(10)))

hist(p, main = "Abundance of species in an ecosystem")

alphas <- seq(0, 5, 0.25)
index <- sapply(alphas, \(a) diversity_index(p, a))

plot(
  alphas,
  index,
  ylim = c(0, 10),
  xlab = "𝛼",
  ylab = "Index",
  main = "Diversity decreases with 𝛼"
)
abline(1 / max(p), 0, lty = "dashed")

With a bit of rearranging, we can see that is the reciprocal of the generalized mean of with these same values as weights

Formulating as a generalized mean shows a clear link between diversity indexes and concentration indexes, as is the reciprocal of the well-known Simpson index (or Herfindahl–Hirschman index if you’re an economist). (It also shows a link with measures of entropy, as is a generalization of Shannon entropy.) What makes a diversity index different from a measure of concentration (or entropy) is that it expresses diversity in terms of the effective size of the ecosystem that would give rise to a particular concentration of species if species were all equally abundant. Intuitively, gives the effective size of the ecosystem if all species were as prevalent as species . Rather than considering a single species, uses the average abundance across all species to arrive at a measure of diversity.

Decomposing diversity

Hill (1973) notes that different choices for imply different sensitivities to rare versus abundant species in an ecosystem. Setting means that diversity depends only on the number of species, no matter how rare some may be, whereas when then only the most prevalent species influences diversity. We can go one step further by decomposing so that it’s represented as an arithmetic mean of the effective size of each species and we can see the contribution of each species towards total diversity. We’ll do this by borrowing some of the machinery from price and quantity indexes to derive weights such that

The core tool to do this comes from my {gpindex} package.

diversity_weights <- function(x, alpha) {
  gpindex::transmute_weights(alpha - 1, -1)(x, x)
}

When , each species contributes the same amount to overall diversity.

diversity_weights(p, 0) / p

 [1] 1 1 1 1 1 1 1 1 1 1

Increasing to 2, the measure of diversity decreases and more weight is shifted towards more abundant species.

diversity_weights(p, 2) / p

 [1] 0.02204630 0.02298373 0.04003915 0.04733940 0.05467209 0.07962440
 [7] 0.10013170 0.14336078 0.48334916 1.48242257

As alpha becomes larger, rare species get a small weight and contribute little towards overall diversity.

diversity_weights(p, 5)

 [1] 4.429620e-05 4.817851e-05 1.481826e-04 2.083463e-04 2.795183e-04
 [6] 6.049503e-04 9.729488e-04 2.068354e-03 3.261787e-02 9.630074e-01

diversity_weights(p, 5) / p

 [1] 0.004974804 0.005190128 0.009163420 0.010897035 0.012658720 0.018811300
 [7] 0.024058228 0.035722328 0.167085953 1.608432485

This gives a different way to view the prevalence of a species, not just by their abundance but by how their abundance contributes towards overall diversity in an ecosystem.