Skip to content

Layer 4: Learning System

Make the truth observable, then run disciplined experiments against it. This is the layer that turns marketing from guesswork into a structured practice. It has two halves: measurement (can we see what is happening) and experiments (how we test and decide). See the system index for how this layer fits the whole.

Research lineage

Sean Ellis, Hacking Growth. Growth is high-speed, cross-functional experimentation: rapidly test ideas, discard what fails, and scale what works, all aimed at moving the North Star and its inputs.

Measurement

Source-of-truth notes

  • The occupancy flow is solid. Move-ins, move-outs, and occupancy reconcile exactly against analytics.occupancy_monthly when computed with its definitions (a move-in is dated by COALESCE(start_time, billing_start_date), a move-out by COALESCE(cancel_at_period_end, cancelled_at), counting only MONTHLY reservations with a subscription and a recurring payment; the Salem diagnosis has the worked method). Validate any flow analysis by recomputing occupancy and matching the view before concluding.
  • Ad spend is solid. Platform-reported spend and structure live in the PostHog warehouse (googleads.*, metaads.*), queryable by SQL. Numerics are strings; cast them. Campaigns can be filtered to a market by name (e.g. "Salem", "Cherry City").
  • Tours are unreliable historically and may stay that way. A tour is required for every lockout move-in, so true tours ≥ move-ins each month; use move-ins as the tour floor. Recorded tour counts are undercounted and backfills cannot reach reliably back many months, so the historical close rate is unknowable. The ~80% close is a community-manager anecdote, not a measured number. It becomes measurable only going forward, and only if tours are logged reliably from here.
  • Lead volume is captured; the linkage to tours and move-ins is not. The attribution stack rolled out in waves (first row 2026-04-12; CAPI ~Apr 27; tour attribution ~May 10; Meta Instant Form sync ~Jun 15), so thin history before mid-April is by design. Past that, the gap is structural, not rollout, and it sits upstream. The attribution chain is wired end to end: tours (ReservationCreationService) and move-ins (FulfillmentService/PaymentIntentHandler write/carry MONTHLY), and #727 already broadened carry-forward to inherit from any prior attribution. It produces near-nothing because there is nothing to inherit: tour-to-channel is dark (1 of ~25 recorded tours carries a channel; tours from DM/text/walk-in backfills have no attribution), so the move-in carry-forward inherits an empty chain (3 MONTHLY rows ever). Lead volume is fine going forward (Meta lead action in the warehouse has real history back to February; the meta_leads table only syncs ~recent, so prefer the warehouse; on-site inquiries since March). So we can read spend and lead volume by channel, but not a channel's leads through to a signed lockout. The fix is upstream: populate tour attribution, and the existing move-in carry-forward inherits it. That fix is live as of 2026-06-25 (PR 852): a required coarse CM lead-source dropdown on staff-created tours, written only as the last fallback so it never overrides a real signal, plus a verified audit that every member move-in path carries forward. It closes the gap going forward, as CMs enter the source from here, not for past tours, and only as well as that entry discipline holds.

Honest caveat carried from prior work: this layer has the strongest tooling and yet the weakest record of turning observation into wins. When measurement is the focus, re-examine the source of truth rather than assuming the current picture is right.

The scorecard

Settled and standing: scorecard.md holds the canonical rows, sources, queries, and the middle-gap reality; it is read weekly and each snapshot is logged in runs/. Because it spans the prod DB (move-ins, tours, occupancy) and the PostHog warehouse (spend, leads), it is a documented query set, not an automated dashboard. As of 2026-06-25 a live view, analytics.cold_start_scorecard (PR 852 + 853), produces the funnel rows and ad columns (meta_spend_cents, google_spend_cents, meta_ad_leads) from one prod SELECT, joining ad spend to a location by campaign-name/mgId match. The full ad history is backfilled into the raw ad_spend_daily/ad_lead_daily tables (Meta from 2025-09, Google from 2026-01) and the daily cron keeps it current. Caveat: the card surfaces only campaign-name-matched, in-grid spend, so general/regional spend lives in the raw tables, not on a location's row.

Rows: spend by channel → leads by channel (directional) → tours (floor = move-ins) → tour close rate (forward-only) → move-ins → occupancy %, with hourly distinct. Read top (spend, solid) to bottom (move-ins, solid); the middle is directional. The live instance is Salem; baseline (row-zero) is runs/2026-06-25-salem-scorecard.md.

Experiments

The loop

Per the cold-start playbook: analyze, generate ideas, prioritize by ICE (impact, confidence, ease), run one or two, review against the North Star, decide. The discipline for our setting:

  • Low volume means decision rules, not p-values. At a handful of move-ins a month, significance is unreachable. Pre-commit to a minimum count plus a directional threshold and hunt step-changes.
  • Fixed envelope. Reallocate the per-market budget, do not grow it. Test by splitting or shifting spend.
  • One or two levers at a time, sequenced so they do not confound the read.
  • Explore, then exploit. Run for volume early to learn what converts, with downstream cohort measurement as the guardrail; tighten toward quality as converting segments appear.
  • Lean on the reliable signal. Where tours and leads are unreliable, judge experiments on move-ins and occupancy.

Log format

Each run: hypothesis · lever · input it targets · ICE · metric and threshold · decision rule · result · decision. Defined-but-not-run experiments live here too, with result/decision blank.

Running log

# Market Experiment Input Status Result
1 Salem Restore high-intent search; move Meta off OUTCOME_TRAFFIC to leads first visits / move-ins Running (2026-06-25) watching move-ins, ~4-8 wk lag
2 Salem On-ramp: tour vs free hours first visits; visit-to-move-in Defined, queued behind #1 —

Full definitions live in the Salem instance.

Gaps and next steps

  • Done (2026-06-25): scorecard settled (scorecard.md) with a baseline snapshot logged; weekly cadence set.
  • Live (2026-06-25, PR 852 + 853): the middle-gap lever is a coarse CM lead-source dropdown on the staff create-tour flow, written only as the last fallback (the email/phone auto-match against meta_leads was already shipped, #820/#821), plus a verified audit that every member move-in path carries forward (lone exception: the B2B org room, out of funnel), an ad-spend/lead sync cron, and the analytics.cold_start_scorecard view with ad columns joined by campaign-name/mgId match (Phase 2, PR 853). It unblocks the Google-vs-Meta verdict going forward, as CM-entered tours accumulate.
  • Done (2026-06-25): full ad history backfilled into prod (721 spend + 532 lead rows; Meta from 2025-09, Google from 2026-01) and surfaced on the scorecard; the daily cron keeps it current. The one ongoing manual dependency is CM data-entry discipline on the new lead-source dropdown. Surfacing unallocated/regional campaign spend (matched to null today, sitting in the raw tables) is a deferred someday item.
  • Make tours logged reliably going forward so the close rate becomes measurable (historical is lost).
  • Run the weekly scorecard and watch the lane call land in move-ins.