What is markov-regime-features?

This Claude skill provides a specialized technical guide for debugging constant Markov regime features in Reinforcement Learning (RL) observations. It specifically addresses the issue where Hidden Markov Model (HMM) probabilities appear as uniform priors (0.33) instead of dynamic estimates, offering a stateful solution using the InferenceObservationBuilder to ensure accurate volatility and trend detection in financial time-series data.

When should I use markov-regime-features?

markov-regime-features is useful in the following scenarios: • Fixing 'constant horizontal band' artifacts in RL observation heatmaps during model training and evaluation. • Implementing stateful Markov regime tracking to replace stateless functions in financial trading agents. • Configuring GPU-accelerated HMM systems to provide real-time probability estimates for market volatility and trends. • Troubleshooting feature engineering pipelines where HMM-based observations fail to update with price history.

name	markov-regime-features
description	"Debugging constant Markov regime features in RL observations - when HMM probabilities show uniform values instead of dynamic regime estimates"
author	Claude Code
date	2025-12-13

markov-regime-features - Research Notes

Experiment Overview

Item	Details
Date	2025-12-13
Goal	Debug why Markov features (indices 19-24) appeared constant in observation heatmaps while other features showed high variability
Environment	Python 3.10, PyTorch 2.0+, alpaca_trading package
Status	Success

Context

When visualizing RL observation features as a heatmap, the Markov regime features (volatility and trend probabilities) appeared as constant horizontal bands with values ~0.33, while all other features showed expected variability over time. This made the Markov features useless for regime detection.

Root Cause

The standalone build_inference_observation() function defaults to uniform priors when markov_vol_probs and markov_trend_probs are not provided:

# From inference_obs_builder.py:391-401
if include_markov:
    if markov_vol_probs is not None:
        obs[:, feat_idx:feat_idx+3] = markov_vol_probs
    else:
        obs[:, feat_idx:feat_idx+3] = [0.33, 0.34, 0.33]  # Uniform prior

The 6 Markov features are:

Index	Feature	Description
19	vol_prob_low	P(low volatility regime)
20	vol_prob_medium	P(medium volatility regime)
21	vol_prob_high	P(high volatility regime)
22	trend_prob_down	P(downtrend regime)
23	trend_prob_neutral	P(neutral regime)
24	trend_prob_up	P(uptrend regime)

Verified Workflow

Solution: Use InferenceObservationBuilder class

The InferenceObservationBuilder class maintains Markov state and updates probabilities based on price history:

from alpaca_trading.gpu.inference_obs_builder import InferenceObservationBuilder

# Create stateful builder with GPU Markov system
obs_builder = InferenceObservationBuilder(window=100, use_gpu_markov=True)

# Build observation - Markov states are updated from price history
obs = obs_builder.build(
    prices=prices,
    high=high,
    low=low,
)

# Access current regime estimates
print(f"Volatility: {obs_builder.vol_probs}")  # e.g., [0.15, 0.60, 0.25]
print(f"Trend: {obs_builder.trend_probs}")      # e.g., [0.20, 0.30, 0.50]

Visualizing Regime Evolution

To see how regimes change over time, build observations at multiple points:

eval_points = list(range(150, n_bars, 10))
vol_history = {'low': [], 'med': [], 'high': []}

for end_idx in eval_points:
    obs_builder.build(prices=prices[:end_idx], high=high[:end_idx], low=low[:end_idx])
    vol_history['low'].append(obs_builder.vol_probs[0])
    vol_history['med'].append(obs_builder.vol_probs[1])
    vol_history['high'].append(obs_builder.vol_probs[2])

plt.plot(eval_points, vol_history['high'], 'r-', label='P(High Vol)')
plt.plot(eval_points, vol_history['low'], 'g-', label='P(Low Vol)')

Failed Attempts (Critical)

Attempt	Why it Failed	Lesson Learned
Using `build_inference_observation()` directly	No Markov state passed, defaults to uniform [0.33, 0.34, 0.33]	Use `InferenceObservationBuilder` class for stateful Markov tracking
Setting `include_markov=False`	Removes features entirely, breaks model compatibility	Keep Markov features, just ensure proper initialization
Manual Markov probability calculation	Complex HMM implementation needed	Use `GPUMarkovSystem` via `use_gpu_markov=True`

Final Parameters

# Correct usage for dynamic Markov features
obs_builder = InferenceObservationBuilder(
    window=100,           # Must match training window
    use_gpu_markov=True   # Enable GPU-accelerated HMM
)

# For live trading - maintain single builder instance across bars
# For backtesting - create new builder per symbol, maintain across timesteps

Key Insights

The standalone function is for single-shot inference where Markov state is passed externally
The builder class is for stateful inference where Markov state evolves with price data
GPU Markov system uses last 20 returns to estimate regime probabilities
Uniform priors [0.33, 0.34, 0.33] indicate Markov system not receiving updates
In heatmaps, constant horizontal bands at indices 19-24 are a red flag for this issue

References

alpaca_trading/gpu/inference_obs_builder.py - Feature building implementation
alpaca_trading/gpu/harmonized_gpu.py - GPUMarkovSystem class
notebooks/develop_branch_testing.ipynb - Visualization examples

markov-regime-features

When & Why to Use This Skill

Use Cases