Statistical Tools for Performance Monitoring

A Lot of Graphs

Resolution

Tools

• collectd
• statsd
• graphite

Kale Stack

• github.com/etsy/skyline
• github.com/etsy/oculus

Skyline

Skyline Internals

• Horizon agent
• Redis
• Analyzer agent
• Flask (Python) Web App

Skyline Algorithms

• median absolute deviation
• grubbs
• first hour average
• stddev from average
• stddev from moving average
• mean subtraction cumulation
• least squares
• histogram bins
• ks test
• second order anomalies

Test Subject

Test Subject

Test Subject

EWMA

exponentially-weighted moving average

Distribution Change

Distribution Change

Oculus

Oculus Internals

• Skyline Import Script and Cronjob
• Resque workers
• ElasticSearch
• Sinatra (Ruby) Web App

Finding Similar Graphs

• correlation (Pearson, Spearman)
• Euclidean distance
• dynamic time warping (DTW)
• discrete Fourier transform (DFT)
• discrete wavelet transform (DWT)

Cluster Centers

Cluster Members

Two Weeks

Holt-Winters

triple exponential smoothing

• needs a lot of data
• sensitive to outliers
• can't handle 3 seasons + holidays
• overfitting

Time Shifting

Q&A

Anton Lebedevich

mabrek@gmail.com

twitter.com/widdoc

github.com/mabrek