Applied Statistics for Performance Monitoring

A Lot of Graphs

Contents

Real Data vs. Ideal Models

Outlier vs. Changepoint

Outlier vs. Changepoint

Outlier vs. Changepoint

Resolution

  • >=5min
  • 1min
  • 10s
  • <=1s
  • Load Testing (Tuning)

    Transient Response

    Failure on Target Metric

    Failure on Target Metric

    Failure on Target Metric

    Filtration

    Missing or Constant

    Changed Mean

    Nonlinear

    ndiffs: diff until kpss says it's stationary

    Production Classics

    Test Subject

    Test Subject

    Test Subject

    Moving Average

    EWMA

    exponentially-weighted moving average

    Control Charts

    Two Weeks

    Holt-Winters

    triple exponential smoothing

    Time Shifting

    Production Experimental

    Autocorrelation

    Autocorrelation

    Ljung-Box Test

    Distribution Change

    Distribution Change

    Distribution Change

    Distribution Change

    2-Sample Tests: Good

    Kolmogorov–Smirnov, Cramér–von Mises

    2-Sample Tests: Bad

    Kolmogorov–Smirnov, Cramér–von Mises

    Finding Similar Graphs

    Cluster Centers

    Cluster Members

    Cluster Members

    Clustering

    Tools

    R

    add.smooth <- function(m) {
      r <- nrow(m)
      ms <- sapply(m, function(y) {
        ave(coredata(y),
            seq.int(r) %/% max(3, r %/% 150),
            FUN=function(x) {mean(x, na.rm=T)})
      })
      df <- data.frame(index(m)[rep.int(1:r, ncol(m))],
                       factor(rep(1:ncol(m), each = r), levels = 1:ncol(m)),
                       as.vector(coredata(m)),
                       as.vector(coredata(ms)))
      names(df) <- c("Index", "Series", "Value", "Smooth")
      df
    }

    Kale Stack

    Skyline

    image from github.com/etsy/skyline

    Skyline Internals

    Skyline Algorithms

    Oculus

    image from github.com/etsy/oculus

    Oculus Internals

    Q&A

    Anton Lebedevich

    mabrek@gmail.com

    twitter.com/widdoc

    github.com/mabrek

    /

    #