[{"body":"One of my biggest commercial wins was a seemingly small UI tweak that led to a roughly 2% marketplace revenue bump. Teachers Pay Teachers (TPT, think \u0026quot;Etsy for Teachers\u0026quot; -- educational resources created by and sold to other educators) was a mature marketplace, where revenue wins were typically under 0.5%. How we got this win I think is instructive, covering the value of a search scientist on a search team, and a holistic view of search UI/UX and algorithms.\nRatings and reviews are important proof of value to users searching on an e-commerce site. In 2020, the resource cards on the search results page looked like this -- the average historical rating, shown as partial stars, followed by the number of ratings in bold text.\nReconstructed historical design of a resource card on the TPT SERP, showing actual rating as stars, and bolded rating count. The Problem We knew from user interviews that buyers anchored on the number of ratings as a critical signal of value, and that the partial star format made it hard to see differences. We also used the number of ratings as a component of search ranking -- everything else being equal, resources with more ratings were ranked higher.\nAt some level this is reasonable. Historically popular resources were generally good. But this pair of design features -- a UI design that focused on rating count, and an algorithm that reinforced that -- had serious down-sides. First, it clearly penalized new resources, which may have been excellent. Second, it meant that resources whose first few ratings were poor, for whatever reason, were basically doomed. Nobody will purchase an item with \u0026quot;⭐️⭐️ (2)\u0026quot; next to it.\nThe Approach So, we set about trying to solve these problems. The goal was multi-part:\nBuyers should anchor on how good the resource likely is, not how historically popular it has been. Sellers should not be penalized for bad luck on their initial ratings. The UI should be clear and informative. The solution we landed on had several components. Critically, all of those components had to be deployed together. Let me walk you through them:\nOn the front end, resource ratings were shown numerically, with the rating in bold and the less-important count now shown in grey. On the back end, a statistical model estimated the likely long-term rating average for this resource, based on the data accumulated so far. (More on this below.) In the search engine, the likely long-term rating average replaced the rating count in the scoring algorithm. Now, the page looks like this:\nCurrent design of a resource card on the TPT SERP, showing estimated rating and grey rating count. Note that, as a buyer, you're likely to focus on the highlighted number, which now reflects our best guess of where the ratings will eventually land. This is a better signal of quality. And, you'll now see more items with high ratings, even if they're newer and haven't had the opportunity to accumulate many ratings yet.\nThe Algorithm Consider two resources. Resource 1 has 200 ratings, averaging 4.7 stars, and is from a seller whose resources average 4.6 stars. Resource 2 only has 2 ratings, of 3 and 5, averaging 4.0 stars. But this resource is from a seller whose many resources average 4.8 stars.\nHow should all of this information be put together to construct an estimate of the long-term rating average, assuming many people will eventually purchase and rate the item?\nThe solution is Bayesian statistics. The intuition is not too complicated -- if you don't have much (or any) data, \u0026quot;borrow\u0026quot; some data from other cases until additional data comes in. So, for Resource 2, you can supplement the existing 2 ratings with some number of ratings of other resources by the same seller, or if necessary from other sellers, to get a more conservative estimate of the eventual rating.\nHere's a very simple simulation of this in R code. I'm papering over a number of technical details, but assume that we've established that we want to borrow 50 observations from other resources. This code computes the expected value of the rating for the two resources. You can see that Resource 2 has an expected value that's quite different from the 4.0, while Resource 1, with many more actual ratings, changes little.\n1# Each resource has an actual number of ratings and an actual average 2# rating. Each is also linked to a seller whose *other* resources have 3# their own average rating -- this is the \u0026#34;prior\u0026#34; belief about a 4# typical rating for this seller, before we\u0026#39;ve seen much data on this 5# particular resource. 6resources \u0026lt;- data.frame( 7 resource = c(\u0026#34;Resource 1\u0026#34;, \u0026#34;Resource 2\u0026#34;), 8 n_ratings = c(200, 2), 9 actual_rating = c(4.7, 4.0), 10 seller_rating = c(4.6, 4.8) 11) 12 13# We \u0026#34;borrow\u0026#34; this many imaginary ratings from the seller\u0026#39;s other 14# resources. The fewer real ratings a resource has, the more its 15# estimate will be pulled toward the seller\u0026#39;s average. 16# Tuned as part of the model-fitting process, but static in production. 17borrowed_n \u0026lt;- 50 18 19# The posterior (Bayesian) estimate is just a weighted average of the 20# resource\u0026#39;s actual ratings and the borrowed, seller-level ratings, 21# weighted by how many ratings each group contributes. 22resources$posterior_rating \u0026lt;- with(resources, 23 (n_ratings * actual_rating + borrowed_n * seller_rating) / 24 (n_ratings + borrowed_n) 25) 26 27resources 1## resource n_ratings actual_rating seller_rating posterior_rating 2## 1 Resource 1 200 4.7 4.6 4.680000 3## 2 Resource 2 2 4.0 4.8 4.769231 Tuning this algorithm to work properly with 1-5 star ratings, handling sellers with few other ratings, and other edge cases, requires careful analysis by a statistically-savvy data scientist. From there, computing these values nightly, and incorporating them into the search algorithm, requires some data pipelines to be set up but is not particularly difficult.\nflowchart LR PDB[(Production DB)] IDX[Nightly Indexing Job] ALG[(Algolia Index)] BE[Backend] FE[Frontend / SERP] IDX --\u003e|builds per-resourcerating estimates| PDB PDB --\u003e|resource data + estimates| IDX IDX --\u003e|computes tiebreakerscore, incl. estimates| ALG ALG --\u003e|search results| BE PDB --\u003e|resource details| BE BE --\u003e|API| FE classDef changed fill:#ffe08a,stroke:#c98a00,stroke-width:2px; class PDB,IDX,FE changed; The Impact We ran an A/B test of these changes, and saw a roughly 2% increase in sales during the rollout, with neutral or positive impact on other metrics. The rollout required careful communication from the product marketing team, as some sellers saw shifts in how often their historically popular resources appeared, but we were able to clarify the situation and complete the rollout.\nOverall, this example shows how search design and algorithms are tightly coupled. Changing the algorithm without the UI would have confused users -- items with large bold rating counts would seem wrongly ranked for no visible reason. Changing the UI alone would have been equally problematic, highlighting a noisy metric that penalized bad luck. Together, the changes directed users to a useful signal, encouraging them to click through and make the qualitative evaluation critical for purchase.\nNotes It's been a few years since these changes were made. I may have mis-recalled some details. Other aspects of the page design have subsequently evolved, and I have no information as to the current algorithm. Statistical note -- The algorithm was an empirical Bayes shrinkage model with a normal approximation to the ordinal data. Offline back-testing allowed computation of variances and the shrinkage parameter. Production code was only slightly more complex than what's shown here. This article was primarily authored by me, with secondary contributions from AI systems. I wrote almost all of the text. The AI generated the R code (which I reviewed) and the diagram, and made a number of suggestions and copy-edits. Are you looking for support with search and discovery on your company's website? Does this post make you think I could help your company make great design or technology decisions? I'm a freelance consultant with extensive experience -- please reach out! ","date":1783555200,"id":"0","link":"/2026/07/big-win-small-search-ui-change-predictive-model/","section":"post","summary":"One of my biggest commercial wins was a seemingly small UI tweak that led to a roughly 2% marketplace revenue bump. Teachers Pay Teachers (TPT, think \u0026quot;Etsy for Teachers\u0026quot; -- educational …","tags":["search","ux","predictive-model","machine-learning"],"title":"A Big Win from a Small Search UI Change and a Predictive Model"},{"body":"A couple of months ago I released Almost Five, a fuzzy-text watchface for the reborn Pebble smartwatch. It's been fun hacking on watch faces, but I realized I had a practical gap: I needed a count-up stopwatch I could wear on my wrist while cooking, without pulling out my phone.\nThere are lots of timer apps on the Pebble App Store, but they're mostly focused on count-down scenarios, where you know exactly when the timer needs to end. But sometimes, especially in cooking, you want to know how long it's been, so that you can adjust as you go. How long has this steak been on the grill? How long have I been reducing this sauce? I don't see great stopwatch-like designs for that use-case.\nSo I built One More Minute, a count-up timer app with a twist -- it vibrates the Roman numeral of each completed minute on your wrist. The regular vibrations keep you in tune with the passage of time, with minimal disruption of your flow or attention.\nTwo timers, one vibe assignment The app displays two independent count-up timers in stacked zones. Each zone shows:\nThe elapsed minutes as a large center number A seconds progress bar that fills left-to-right over 60 seconds A play/pause indicator (filled triangle when running, two bars when paused, blank at zero) A vibration icon showing which timer will vibrate every minute At most one timer at a time can vibrate. You cycle the assignment with the SELECT button: timer 1 → timer 2 → none → timer 1.\nThe Roman numeral trick Here's the fun part. When a timer has the vibe assignment enabled, it emits a vibration pattern encoding the Roman numeral of the completed minute:\nI = short pulse (125 ms) V = medium pulse (250 ms) X = long pulse (500 ms) So minute 8 (VIII) vibrates as: medium, pause, short, short, short. Minute 14 (XIV) is: long, pause, short, medium. The gaps between symbols are tuned for perceptual clarity -- 100 ms between identical symbols in the same group, 350 ms between different symbol groups.\nThe patterns go up to 39 minutes (XXXIX), which is the practical ceiling for this small vibration engine. Beyond that, you'd need a different approach. I wrote about the background of this design a decade ago — it's been fun to finally implement a version of it.\nControls Button Short press Long press UP Start/stop timer 1 Clear timer 1 DOWN Start/stop timer 2 Clear timer 2 SELECT Cycle vibe assignment — BACK Exit app — Timers freeze where they are when stopped, so you can pause and resume without losing your place.\nTechnical notes The app is pure C, no phone-side JavaScript. The UI is custom-drawn with layered windows — no framework shortcuts. The code was written in Kilo Code with a mix of Qwen and Claude agents.\nThe vibration engine is the most interesting piece. We wrote detailed specs and tests for the Roman numeral pattern generation, which made it easy to iterate on the timing gaps until they felt right on the wrist.\nTry it out The MIT-licensed source is available on GitHub. If you have a Pebble and the pebble CLI installed, you can sideload it directly:\n1pebble build 2pebble install --phone \u0026lt;PHONE_IP\u0026gt; Or test it in the emulator:\n1pebble install --emulator flint I'm looking for testers and feedback. If you build it and try it out, let me know how the vibration patterns feel in practice.\n","date":1782777600,"id":"1","link":"/2026/06/one-more-minute-pebble-timer/","section":"post","summary":"A couple of months ago I released Almost Five, a fuzzy-text watchface for the reborn Pebble smartwatch. It's been fun hacking on watch faces, but I realized I had a practical gap: I needed a count-up …","tags":["pebble","smartwatch","timer","design"],"title":"One More Minute: A Pebble Count-Up Timer with Roman Numeral Vibrations"},{"body":"One of my specialties as a freelance consultant is discovery systems for e-commerce -- web pages and backend systems for helping users search for and find what they want or need. For the last year and a half, I've been working as a consultant for DesignShop, a company that sells materials samples for home renovations. Want to find a countertop, cabinet, paint, and tile combo for your kitchen reno that looks great, with samples shipped overnight? Check 'em out.\nMuch of my work at DesignShop has been on ranking and relevance -- finding ways to improve which items appear when you enter a search or apply a filter, and the order in which they're shown. Recently, I got to work on a revamp of the visual design and user experience of the Search Results Page (SERP) -- everything except ranking and relevance! In collaboration with colleagues in design, engineering, and others, we created a simple and straightforward search experience that, we believe, guides users towards good discovery paths. Let me walk you through some key details, some of which are a little unusual, and explain how we got to this design.\nWe had a few constraints. Keep it very simple -- this design was replacing an overly complex search drop-down that was no longer relevant or useful. Help users do the things that will get them a great list of samples to browse efficiently, on both mobile and desktop devices. And minimize future maintenance costs as well as expensive search engine queries.\nThis is what the SERP looks like as of this writing:\nDesignShop search results page, showing numbered features discussed in text. Decisions Let's go through some of the decisions and design elements.\nGet users to the SERP, then refine there Simple search bar in the header. When you click on the header search bar, you see just recent search history and popular item suggestions before you start typing, and autocomplete after you start typing. No search-as-you-type -- previous iteration was visually busy and difficult to refine effectively. The value of showing a few examples when just a few letters have been typed is not high, and there's limited screen pixels to show results, so we made the tradeoff to keep it simple. We want people to type something and press Enter, to get to the SERP, which is where refinement is easiest. Search bar in the header with no input -- just recent history and some popular samples. Note that not every e-commerce site should make this decision. I really like this recent teardown of the HOKA sneakers website. Their comparatively small catalog makes complex drop-downs potentially helpful for users, showing just a few results without a full page load.\nMove attention from the header search box to the search results and refinement. We want users to refine their query when results are too broad or off-target. So we placed the editable search box next to the results to make that relationship explicit. On the SERP, we reduced the site-wide header search box to just an icon. We think this signals \u0026quot;start over\u0026quot; rather than \u0026quot;refine this result set.\u0026quot; Many SERP designs keep the editable box in the header, but for DesignShop, with a substantial site-wide header that includes category page and special features links, this pattern is clearer. Make filter applications obvious and cheap Keep filters visible on desktop, easily accessible on mobile. The filter accordion is always visible on desktop, on the left of the search results, and is easily accessible on mobile. We want people to filter, because structured filters usually outperform raw keyword matching on multi-word queries (color:pink category:tile often yields better results than does pink tile). DesignShop does do some automatic filtering and reranking, but coverage is imperfect. Keeping filter categories visible also teaches users what the catalog contains, critical with tens of thousands of items, in dozens of brands and categories, where the point of the site is to purchase samples from multiple categories to validate that they'll go well together. The tradeoff to keeping the filters visible is fewer pixels to showcase results, but on desktop this is still the right balance.\nSearch results on mobile. Tell users exactly what to do: refine the query. Scrolling helps users sample variety, but with thousands of results it is rarely the fastest path to a good choice. We explicitly prompt refinement so users lean on categories and item properties instead of endless browsing.\nUse Post-Query Refinement Suggestions (PQRS). As I previously described here, these are high-value filter buttons elevated from the sidebar when an algorithm suggests that they will most effectively narrow large result sets.\nMake the results grid legible, informative, and trustworthy Secondary results are distinct but inline. An early version of the SERP interleaved curated Collections with sample results in a way that made evaluation and visual scanning difficult for users. The current version still shows both, but visually demarcates Collections results so users can, if they want, focus on them without confusing them with primary product matches.\nShow merchandising tags to explain value. Labels like \u0026quot;Best Seller\u0026quot; provide proof-of-value cues and help users understand that ranking reflects meaningful signals.\nDiversify brands in the top row. Just as we draw attention to filters to educate users about what's available, we sometimes use search ranking to highlight the diversity of product brands on the site. DesignShop lightly reorders the first few items so users are more likely to see multiple brands early in their review of the results.\nThere's always more to do: results to measure, tradeoffs to balance, tweaks to fix, and new hypotheses once the metrics and real user feedback arrive. A search design never really finishes -- it becomes the baseline for the next set of opportunities to help the user. Even so, what shipped already feels like progress: a simple path from the header into the SERP, refinement sitting next to the result set instead of competing with it, and a grid that keep the results legible to searchers. I think that's a solid step forward.\nNotes:\nThis post is my personal take on the redesign, and shouldn't be read as an official DesignShop document. As a technical aside, with this re-write, we observed that Algolia's InstantSearch library has reduced value now, in the age of AI assisted engineering. It's just so fast to write code, if the requirements are clear, that the library adds opaque layers compared with calling the API directly and updating the page. We ended up stripping it out of the SERP (although it's still used elsewhere on the site). This post was human-authored with AI used for feedback and light editing. Of course, most of the software implementation was done by AI agents, supported by human engineers. ","date":1778198400,"id":"2","link":"/2026/05/design-decisions-new-search-results-page/","section":"post","summary":"One of my specialties as a freelance consultant is discovery systems for e-commerce -- web pages and backend systems for helping users search for and find what they want or need. For the last year and …","tags":["search","ux","product","e-commerce"],"title":"Design Decisions for a New Search Results Page"},{"body":"You might remember the old Pebble smartwatch -- a minimalist e-ink watch that was popular among a certain set ten or so years ago. Well, it's back -- the original creator is manufacturing new hardware, and the software has been open-sourced. I wasn't on the bandwagon when they were originally popular, but I picked up the old/new Pebble 2 Duo recently and have been enjoying it.\nRecently I decided to start building apps for it. I have a handy app for putting my grocery list on my wrist, but it's based on a reverse engineered API, which the grocery list company doesn't support, so I can't publish it. What I can publish though is an open-source watch face that I've been hacking on. Based on a couple of iterations of previous work by others (PebbleTextWatch and Fuzzy-Text-Watch-Plus), both last touched in 2016, and with the assistence of an AI agent, I've now released Almost Five!\nThe watch spells out the time, just as you'd normally say it in conversation. Ten til Twelve, Almost Three, etc.\nThe previous developers made a great start, but I've made a number of enhancements:\nAdded informal American English, as well as French and (romanji) Japanese. (Generating translations is now literally the easiest thing in the world to do.) Added the day of month to the bottom of the screen. Added no-bluetooth and low-battery indicators. Added support for iCal calendars to remind you \u0026quot;Meeting Soon!\u0026quot; Improved layout algorithm. Improved development workflows. It's been a fun little side project over the past few days. If you have an old (or new!) Pebble watch, please try it out. I need testers!\nThe app is listed on my software page and the MIT-licensed source is available on github.\n","date":1777507200,"id":"3","link":"/2026/04/introducing-almost-five-newish-pebble-watchface/","section":"post","summary":"You might remember the old Pebble smartwatch -- a minimalist e-ink watch that was popular among a certain set ten or so years ago. Well, it's back -- the original creator is manufacturing new …","tags":["pebble","smartwatch","watchface","design"],"title":"Introducing Almost Five, a New-ish Pebble Watchface"},{"body":"This post is a little different. I have a layperson's/nerd's interest in cosmology and physics, and have tried to read a few books about current research. Recently, I ran across some new work suggesting a pretty radical shift in theory. Not finding a great summary for the non-physicist reader, I had Google Gemini write one for me, in the style of pop-science journalism like Science News. The result, below, is 90% AI's writing, unlike most of what I write these days. I served as the editor, helped it with a few rewrites and framing, and double-checked/fixed the citations. (Gemini tended to hallucinate the wrong journal, but got the content mostly right.) It's possible that there are misunderstandings of the research direction, but I think it's pretty close. And it's an interesting set of hypotheses!\nAre Black Holes the Answer to Dark Matter and Dark Energy? By Google Gemini April 18, 2026\nModern physics is defined by three colossal, open questions that refuse to be solved. These mysteries define the very structure of the universe, yet they are traditionally studied as isolated, unrelated problems. But a series of data-driven breakthroughs between 2025 and early 2026 is driving the community toward a shocking unified conclusion: these three puzzles might not be separate at all. They might be a single \u0026quot;story\u0026quot; of the quantum vacuum, viewed from different scales.\nHere are the three great unresolved threads of the cosmos:\nDark Matter: What is the invisible \u0026quot;stuff\u0026quot; that holds galaxies together? Dark Energy: What is the negative pressure pushing the universe to expand faster? Black Holes: What is the true nature of their nonsingular cores, and how is information conserved when matter falls into them? The starting gun for this potential unification was fired in early 2025 by the Dark Energy Spectroscopic Instrument (DESI). By mapping 11 billion years of cosmic history, researchers found that dark energy is not, as Einstein assumed, a constant property of space. Instead, it evolves over time. Crucially, this \u0026quot;wobble\u0026quot; in dark energy’s strength tracks perfectly with the timeline of when massive stars died and formed black holes. This \u0026quot;timing\u0026quot; is the smoke from a massive gun.\nIllustration of how a non-singularity explanation for black holes might account for dark matter and dark energy. Author: Google Gemini. The First Thread: The Ghost in the Dark (Dark Matter) The problem of Dark Matter is well known: galaxies are rotating so fast that, based on the matter we can see (stars, gas), they should be flying apart. Something unseen -- amounting to roughly 85% of all matter -- is providing the extra gravitational \u0026quot;glue.\u0026quot;\nFor decades, we have searched for new subatomic particles, like WIMPs (Weakly Interacting Massive Particles) or axions, but experiments have come up empty. The new trifold hypothesis re-examines a classic idea that was once discarded: Primordial Black Holes. These are black holes that may have formed instantly during the Big Bang, long before stars.\nThe new hypothesis is that these primordial black holes didn't stay small. They have been \u0026quot;coupled\u0026quot; to the expansion of the universe for 13.8 billion years. This Cosmological Coupling (CCBH) causes them to grow in mass (\\(M\\propto a^{k}\\), where \\(a\\) is the scale of space, and \\(k\\) is a constant), without needing to \u0026quot;eat\u0026quot; regular matter. By 2026, models are showing that if these \u0026quot;coupled\u0026quot; black holes were the initial dark matter, they would have \u0026quot;fattened up\u0026quot; over billions of years, perfectly accounting for the missing mass we see holding galaxies together today. Dark Matter, in this view, is not a new particle; it is the first generation of \u0026quot;living,\u0026quot; growing black holes.\nThe Second Thread: The Architecture of the Singularity (Black Holes) The next thread started as its own problem: what black holes are inside -- singularities and the information paradox -- pursued largely apart from the missing-mass and accelerating-expansion puzzles. The unification below only comes when that interior picture is folded back into the expanding universe.\nStandard General Relativity predicts that the center of a black hole is a point of zero volume and infinite density: the singularity. This prediction is a failure point; infinite results usually mean your equations are incomplete. Singularities also create the \u0026quot;Information Paradox\u0026quot;: if a black hole evaporates and destroys the singularity, the information that fell into it is deleted forever, violating quantum law.\nThe trifold story resolves this by turning to Asymptotic Safety, a theory pioneered by physicists like Astrid Eichhorn. While standard physics sees space-time as smooth, Asymptotic Safety predicts that at the smallest \u0026quot;Planck\u0026quot; scale (\\(10^{-35}\\) meters), it becomes fundamentally fractal. Just as a jagged coastline maintains its shape whether you view it from a plane or a microscope, space-time at this scale maintains a \u0026quot;fixed point\u0026quot; of stability. It doesn't collapse into a zero-dimensional dot; instead, it is regularized into a finite, stable structure.\nThis fractal geometry replaces the singularity with a smooth, non-singular \u0026quot;bubble\u0026quot; of vacuum energy, often called a GEODE (Generic Dark Energy Object). A GEODE is not a \u0026quot;solid\u0026quot; object like a planet. It is a \u0026quot;knot\u0026quot; in a single universal field. Because it is not a dead end (no singularity), information is never lost; it is stored in the complex, fractal fluctuations of the vacuum.\nThe Third Thread: The Mechanism of the \u0026quot;Free Lunch\u0026quot; (Dark Energy) This is the most critical and complex part of the narrative: How does a local GEODE \u0026quot;knot\u0026quot; influence the entire universe, and where does its mass come from? We must walk through the reasoning step-by-step, addressing the classical objections.\nClassical intuition says a black hole grows only by eating. But if it has a non-singular core of vacuum energy, it must obey a new law: Scale Symmetry. If the universe has no \u0026quot;special\u0026quot; scale at the fundamental (fractal) level, then the laws of the smallest \u0026quot;knot\u0026quot; must respond to the laws of the largest cosmic scale.\nThe Feedback Loop This is the Cosmological Coupling mechanism (\\(M\\propto a^{k}\\)). Imagine the universe as a vast rubber sheet, with black holes acting as \u0026quot;knots\u0026quot; embedded within it. When the entire sheet is stretched (the universal expansion \\(a\\)), the sheet pulls on the knots. Usually, objects are held together by internal forces (like atoms), resisting this pull. But GEODEs are pure geometry; they have no internal structure to resist. They are therefore \u0026quot;coupled\u0026quot; to the expansion. Stretching them requires energy, and that stretching energy is stored within the GEODE core as mass (\\(E=mc^{2}\\)).\nThe Zero-Energy Balance How can a black hole get fat without \u0026quot;stealing\u0026quot; matter from somewhere? The answer lies in the Zero-Energy Universe hypothesis. In curved, expanding spacetime, the Law of Conservation of Energy localizes, but the total energy of the entire universe is actually exactly zero.\nThis paradox arises because, in General Relativity, gravitational potential energy is negative, while the energy of matter and vacuum is positive. When space-time expands, it generates more space, which has more negative gravitational energy.\nTo keep the universe’s total energy balanced at zero, this new \u0026quot;gravitational debt\u0026quot; must be perfectly offset. The universe \u0026quot;pays\u0026quot; for this new gravity by creating an equivalent amount of positive energy. The trifold theory argues that this new positive energy is deposited exactly where gravity is strongest: inside the black hole GEODE cores.\nThis completes the feedback loop:\nExpansion: Space expands (triggered by the cosmic constant \\(\\Lambda\\)). Debt: Stretching increases the negative gravity potential of space. Payment: This \u0026quot;loan\u0026quot; forces black hole GEODEs to increase their mass (via coupling \\(M\\propto a^{k}\\)). Push: The increased GEODE mass exerts more \u0026quot;negative pressure,\u0026quot; which accelerates the universal expansion. Bridging the Paradoxes: The Unified \u0026quot;Story\u0026quot; By connecting the three threads above, we are not claiming a finished theory—only that the pieces can be read as chapters of one draft. Zarikas and Mitra (2025) use renormalization-group (RG) flows—the usual bookkeeping for how interactions change with energy scale—to argue that microphysical pictures of how information might be stored could live alongside the sort of cosmological coupling observers infer when they quote something like \\(k\\approx 3\\). That link is technical and contested; it is a live proposal among specialists, not a community-wide verdict.\nOn the classical gravity side, Cadoni et al. (2023) spell out how nonsingular black holes can couple to an expanding universe (including how mass can track the scale factor in simple models), while Cadoni et al. (2025) build explicit cosmological embeddings of regular black-hole metrics and follow how their apparent horizons behave as the background stretches. Researchers are not marching in lockstep—different groups stress different models, data cuts, and degrees of skepticism—but the overlap is suggestive: it increasingly looks as though one mathematically connected story about horizons, regular interiors, and Hubble-scale dynamics might be taking shape, even though no one would fairly call the case closed.\nThe Skeptical Front and What’s Next Many in the community remain \u0026quot;cautiously skeptical.\u0026quot; The 2025 DESI results currently sit at a statistical significance of roughly 4-sigma. While compelling, it falls short of the \u0026quot;5-sigma\u0026quot; gold standard required for an official discovery. Historians of science remember that \u0026quot;\\(k\\approx 3\\)\u0026quot; could easily be a statistical fluke.\nThe standard cosmological model (\\(\\Lambda\\)CDM) offers a potential alternative: Hidden Accretion. Critics argue that the apparent growth of black holes in \u0026quot;dead\u0026quot; galaxies might be explained by standard physics. We might simply be undercounting the amount of low-density \u0026quot;snacks\u0026quot; (gas or small, hidden star mergers) that these black holes consume over billions of years.\nThe true test will come from observations. The next decade will focus on two \u0026quot;smoking guns\u0026quot;:\nGravitational Waves: Future observatories (like LISA) will look for subtle differences in the ripples of space-time when two nonsingular GEODEs merge, compared to the standard \u0026quot;firewall\u0026quot; merging model. DESI Replication: Telescopes like the Vera Rubin Observatory must confirm whether the \u0026quot;\\(k\\approx 3\\)\u0026quot; dynamic dark energy is a true fundamental constant or a mirage. Citations and Further Reading Eichhorn, A. (2026). Where Some See Strings, She Sees a Space-Time Made of Fractals. Quanta Magazine. https://www.quantamagazine.org/where-some-see-strings-she-sees-a-space-time-made-of-fractals-20260311/ Farrah, D., et al. (2023). Observational Evidence for Cosmological Coupling of Black Holes and its Implications for an Astrophysical Source of Dark Energy. The Astrophysical Journal Letters. https://iopscience.iop.org/article/10.3847/2041-8213/acb704 Croker, K. S., \u0026amp; Weiner, J. L. (2019). Implications of Symmetry and Pressure in Friedmann Cosmology. I. Formalism. The Astrophysical Journal 882, 19.\nhttps://doi.org/10.3847/1538-4357/ab32da\nKohler, S. (2019). See also Selections from 2019: Connecting the Universe’s Large and Small Scales. Zarikas, V., \u0026amp; Mitra, A. (2025). Cosmologically Coupled Black Holes and Asymptotic Safe Gravity. Proceedings of the Corfu Summer Institute 2024, School and Workshops on Elementary Particle Physics and Gravity.\nhttps://ui.adsabs.harvard.edu/abs/2025eppg.confE.224Z/abstract Cadoni, M., et al. (2023). Cosmological coupling of nonsingular black holes. Journal of Cosmology and Astroparticle Physics 2023 (11), 007.\nhttps://doi.org/10.1088/1475-7516/2023/11/007 Cadoni, M., Pitzalis, M., \u0026amp; Sanna, A. P. (2025). Apparent horizons in cosmologically-embedded black holes. Journal of Cosmology and Astroparticle Physics 2025 (02), 051.\nhttps://doi.org/10.1088/1475-7516/2025/02/051 ","date":1776470400,"id":"4","link":"/2026/04/black-holes-dark-matter-dark-energy/","section":"post","summary":"This post is a little different. I have a layperson's/nerd's interest in cosmology and physics, and have tried to read a few books about current research. Recently, I ran across some new work …","tags":["cosmology","physics"],"title":"Are Black Holes the Answer to Dark Matter and Dark Energy?"},{"body":"Say you're an Engineering Manager or a Product Manager, responsible for a business-critical search system on a website. Maybe e-commerce, maybe a marketplace, maybe something else. Search is sort of working, it doesn't crash, and it's fast, but there's a feeling that \u0026quot;search isn't great\u0026quot;, and you've been tasked with finding someone to help. Your first instinct, or that of your VP, might be some version of: “we need to hire a search engineer.”\nSometimes that’s true. If your search system is broken at the infrastructure level, you need engineering help immediately. If it tends to crash during high-traffic sales, focus on that problem first. If you need new data pipelines to pull catalog information into your search engine, that's an engineering problem.\nBut for most established products, those aren't the main bottlenecks. Unless you're at a household-name company, the hard part is not getting search to run at your scale. The hard part is getting search to improve in a reliable, measurable way.\nI think many teams are actually looking for a search scientist: someone who can research and diagnose failure patterns in user behavior, design meaningful experiments, build the right internal tools, and lead the application of insights to product and algorithmic improvements.\nWhat Changes After Search Is \u0026quot;Working\u0026quot; Early in a search product’s life, engineering dominates. You need indexing, retrieval, ranking, query understanding, scalable infrastructure, and all the reliability basics.\nAfter that foundation is stable, most production changes become incremental:\nparameter changes to improve relevance and ranking ingestion of new data fields A/B tests of UI design changes tooling to support operational thumbs on the scale Those are important, but they usually aren’t technically difficult to implement, and they don't take many lines of code. The hard part is figuring out what to measure, what needs to improve, what often-small changes might address the opportunity, and figuring out if the change actually worked, or caused unintended side effects.\nThat is search science. It's not just engineering or product work, it's some of each, plus a strong quantitative research core. This builds on Daniel Tunkelang's point that search leadership requires blended product and engineering judgment.\nWhere The Work Actually Is In mature systems, much of the coding work moves away from user-facing production code and into internal tools and analysis code.\nYou need software that helps the team think:\nquery replay and side-by-side result comparison segment-level metric drill-down dashboards annotation and judgment workflows -- making it easy for domain experts to flag issues for follow-up search failure databases and curation tooling experimentation readouts with guardrails offline evaluation and diagnostic notebooks Without these, teams operate on anecdotes and intuition. With these, teams can move from “search feels off” to “high-intent accessory queries on mobile are converting less-well because synonym expansion is overfiring for brand modifiers.”\nWhy This Looks More Like Data Science Than Software Engineering Search quality is fundamentally an inference problem under uncertainty and complex constraints. User intent is messy. Outcomes are sparse and ambiguous. Behavior changes in response to your product. Metrics are proxies. Everything is confounded.\nThat environment needs people who are strong at:\nhypothesis design causal reasoning metric and guardrail construction segmentation and bias checks statistical judgment under noisy data These are core data science skills.\nSenior engineers are still essential, especially for architecture, performance, reliability, and implementation quality. PMs are still essential for prioritization, alignment, and strategy. (See my earlier post on search teams and the product trio.) But if you are asking “who can represent the voice of the user in a search system?”, the answer is a person who can read the behavioral evidence and reason about algorithmic cause and effect.\nIn mature search products, leverage comes less from expanding production engineering capacity and more from strengthening the insight-to-experiment loop. That loop is usually owned best by a search scientist.\nThe Role of AI and Agentic Engineering The LLM-aided revolution in software engineering since 2025 changes the equation even further. I've argued for years that a data scientist should be able to write backend software as well as a software engineer two standard levels below their title seniority, e.g., a Staff Data Scientist should code like a Senior Software Engineer. But what can those people do now, when the AI writes the code under human supervision? Your search scientist can draft front-end changes that work hand-in-hand with algorithmic changes. Or write internal tools in an afternoon, to amplify their own or the team's velocity and improve workflows.\nYou still need a software engineer on a team, to focus on code scalability, robustness, uptime, architecture, and integrations. Not to mention to be on-call if something breaks in production. But the long pole is now even more decision-making, research, and prioritization, not lines of code.\nHow to Hire a Search Scientist If you’re a PM or EM staffing a search team, here is a starting point for the \u0026quot;What You'll Do\u0026quot; section of your job description (and check LinkedIn for current postings):\nOwn search quality diagnosis: query segmentation, failure taxonomy, and root-cause analysis. Design and run experiments: offline eval plans, online tests, guardrails, and reports. Build internal relevance tooling: replay, side-by-side diffing, annotation workflows, diagnostics. Translate findings into shipped changes with PM + engineering partners. Contribute to production code. Maintain a search learning system: metric definitions, experiment history, and reusable playbooks. Talk to and learn from customers as well as internal domain experts. You're looking for someone who can build this capability and run it reliably, again and again.\nNotes:\nAre you looking for a fractional or interim search scientist? Want external help hiring a full-timer? I'm a freelance consultant with extensive experience in this area -- please reach out! The first draft of this post was written by an AI, based on a few notes about the argument to be made, but it was then extensively rewritten and refined by hand. The AI was a secondary author and most ideas and the final voice are mine. ","date":1776038400,"id":"5","link":"/2026/04/you-dont-need-a-search-engineer-you-need-a-search-scientist/","section":"post","summary":"Say you're an Engineering Manager or a Product Manager, responsible for a business-critical search system on a website. Maybe e-commerce, maybe a marketplace, maybe something else. Search is sort of …","tags":["search","product","data-science","management"],"title":"You Don't Need a Search Engineer; You Need a Search Scientist"},{"body":"Software I build small, donationware tools that others may find useful. This page what I've built so far.\nOpTree A small web app for building and sharing Opportunity-Solution Trees.\nOpen OpTree.\nAnnouncement blog post.\nAlmost Five Watchface A Pebble watchface that says it's \"Almost Five\" or \"Seven Twenty\", along with the spelled-out date and other features.\nAlmost Five on the Pebble Appstore.\nAnnouncement blog post.\nOne More Minute A count-up timer app that vibrates the Roman numeral of each completed minute on your wrist. Keeps you in tune with the passage of time.\nOne More Minute on the Pebble Appstore.\nAnnouncement blog post.\nSupport this work If you find these tools helpful and want to support ongoing development, you can do so here:\nBuy Me a Coffee\n","date":1775883600,"id":"6","link":"/software/","section":"","summary":"Software I build small, donationware tools that others may find useful. This page what I've built so far.\nOpTree A small web app for building and sharing Opportunity-Solution Trees.\nOpen OpTree. …","tags":[],"title":""},{"body":"I grew up playing Scrabble -- my grandma was cuthroat. I've played occasionally through the years, and more recently I've been playing Scrabble Go, the official, licensed Scrabble app, with friends. It's fine, but it's full of ads (unless you pay them), it has annoying side games (gems? why am I earning gems?), and I wished it had features like word definitions and better game history. So, starting last year, I built my own web app. I've had half a dozen friends and family members playing with me, each other, and a robot for the last few months, and it's been a lot of fun. I thought I'd share a few things I've learned along the way.\nThe game is a web app, a full-stack TypeScript project (React, Express, PostgreSQL) that I built almost entirely with AI assistance. Here's a few screenshots to give you a feel:\nScrabble game board, showing opponent's play, draft user play, reaction, and more. Leaderboard, showing categories and fun AI-generated funny-plays feature. Main screen, showing active and past games and overall design. Some things I learned (many of which many other people learned before me!):\nYou can get to 80% shockingly fast. The rest still takes time and focus. I started building the app in mid-2025, on Replit. The basic set of screens, the database, the login flow, a basic robot (for testing, initially), and the initial game-play and rules took just a few days. The AI knew a lot about Scrabble, and built most of the core correctly the first time. It was almost playable, but not quite, and I lost steam and took a break until Christmas. Part of why I stopped was that agentic coding patterns and frontier models weren't mature enough.\nWhen I came back, the tools had improved, I understood them better, and the second push was more productive. I fixed the mobile experience, had the AI do a big refactor for maintainability, and got some other critical pieces working in another week or so, and invited my first tester. It was several more weeks until the UI was polished, remaining bugs (including a play-scoring bug) were squashed, and notifications and invitations were working.\nI also got some specific itches scratched, like a score graph that showed who was ahead at various points in the game -- one of the features I'd wished Scrabble Go had.\nCumulative score graph, showing that I was on par with the bot for a while, then it pulled ahead and won. \u0026quot;Vibe coding\u0026quot; evolved into something more like managing a junior developer. When I started in mid-2025, \u0026quot;vibe coding\u0026quot; on Replit was the cutting edge. \u0026quot;Build me this feature\u0026quot; and hope for the best. By the time I was back developing in December and January, best practices and my workflow had shifted to what people are calling \u0026quot;agentic coding\u0026quot;. I maintain the task list, we plan each ticket in advance, with a detailed technical spec, and then I say \u0026quot;go ahead.\u0026quot; I'm the PM and the supervising Tech Lead, and I spend a lot of time on planning and verification. The AI writes essentially all the code, using patterns such as sub-agents. I make the product and architectural decisions. Better processes for systems that are bigger than a demo.\nPlans and specs are critical for larger systems. LLMs are good at implementation but can't read your mind. Unlike a junior developer, they have no experience using the product, attending all-hands meetings, or absorbing the \u0026quot;why\u0026quot; through osmosis. You have to articulate context explicitly. As the tools gained explicit planning modes, I started using them.\nAI remains bad at many things that make a product good. The AI could build a leaderboard. But when I asked it to suggest categories, they were boring — just variations on \u0026quot;who got the highest score.\u0026quot; The categories that actually make the leaderboard fun (best comeback, funniest word) came from me. The same was true for most UI design decisions: menu layout, button placement, notification design.\nI also had to make taste calls about what kind of experience I wanted to create. I didn't implement word challenges (unlike official Scrabble) because they felt too confrontational for the friendly game I wanted. Scrabble Go and Words With Friends don't have them either. Decisions about notification timing (games move forward, but people don't feel hassled) required taste. Taste is still a human job, and will be for quite some time.\nThe robot (AI opponent) required taste applied to AI behavior, not just UI. Early versions played obscure words too frequently, which felt inhuman in a bad way. I ended up limiting the Easy and Medium robot levels to the most common 5,000 and 50,000 English words, respectively. The Medium level also knows all of the two-letter Scrabble words. Easy and Medium modes also randomly \u0026quot;forget\u0026quot; words, making the bot feel more natural while still playing well enough to be challenging. Hard mode has the full dictionary, and doesn't forget words. These decisions lead to robot games that are fun and challenging at just the right level.\nGaps in your own technical skills can still cause bottlenecks. I'm a pretty lousy front-end engineer. When the AI struggled with pixel-level UI work, I couldn't give it technical guidance to get unstuck. I'd end up pasting screenshots and restating the problem in different ways until it worked. This got easier over time as the models improved, but it's clear: AI-assisted development amplifies your strengths and exposes your gaps. You don't need to be an expert in everything, you don't even need to know the programming language being used, but you need enough conceptual grounding to not get stuck and to be able to offer constructive help.\nSmall design features create a sense of fun. I didn't want to build a general-purpose chat system, but I wanted some emotional connection between players. So I added emoji reactions -- a constrained palette (😂 👎 🤔 😤 🎯 💪 etc.) that both human and robot players can use. The robot picks contextually appropriate responses by reading the board state. It's lighter-weight than a text box and more fun.\nAnother small, fun feature is the login codes. You're used to getting an email or text from your bank with a random set of numbers you can paste in to authenticate. But that's impersonal. The app sends you a random seven-letter word from the official Scrabble dictionary, instead! Why paste in \u0026quot;865903\u0026quot; when you could paste in \u0026quot;QUICKLY\u0026quot;? One user wrote me to say he was amused by this feature -- that's a win!\nSmall audiences are worth building for. Half a dozen people play my Scrabble app. That's not a business, and because of pesky trademark laws, it's going to stay very small. But most of the people I invited are still playing, months later. There's something satisfying about building a thing that a few specific people use and enjoy -- different from building for anonymous scale. You get direct feedback (mostly bug reports), and you can make decisions fast because you know everyone personally. As hobby projects go, this one has been very rewarding.\nNotifications are the unglamorous secret to retention. Although I hope playing Scrabble with me is fun, I'm pretty sure that notifications are what reminds everybody to keep coming back. I started out with browser notifications, and recently switched to email notifications, including a daily digest that also includes nudges to look at the leaderboard feature. SMS notifications require a heavy phone number registration process that I started but didn't want to spend the time to finish.\nTechnical debt is still a thing. I started on Replit because it was the fastest path to a working app. When I outgrew it (cost, dev tooling preferences), I had to migrate everything -- development to Kilo Code, hosting to Render, auth from Replit to email login codes, notifications from Firebase push to email via Resend, plus a new domain and a new database. I wrote a 4-phase migration plan with many manual steps that had to be executed in order. It couldn't be delegated to agents. Some parts were hard to test in dev, so a few steps were implemented live. I'm very glad I incurred the technical debt to get started quickly, and am also glad I paid back the debt by doing the migration.\nYou can't build something this big on vibes alone. I genuinely think everyone with ideas should try building things with AI. The barrier is lower than it's ever been. But I don't think you can build an app this complex -- full-stack, real-time-ish multiplayer, auth, notifications, AI integrations, a platform migration -- without some software development experience. There are too many places where you hit conceptual blocks that require understanding what's actually happening, not just describing what you want. The AI is a force multiplier, but you need something to multiply.\nThis started as a \u0026quot;can I do this?\u0026quot; experiment but shifted to something I want to polish and get right.\nNote:\nThe app is invite only, and I'm not sharing the URL. I know all six of my users and I like it that way. But if you're a close enough friend or relative to have my phone number, text me and I'll send you an invite! This post was primarily human-authored, with AI assistance. Notably, the AI interviewed me and drafted the first version of the list, filling a Secondary author role. The ideas and final voice are mine. ","date":1774742400,"id":"7","link":"/2026/03/nine-things-learned-building-scrabble-app/","section":"post","summary":"I grew up playing Scrabble -- my grandma was cuthroat. I've played occasionally through the years, and more recently I've been playing Scrabble Go, the official, licensed Scrabble app, with friends. …","tags":["software","ai","llms","vibe-coding","side-projects"],"title":"Nine Things I Learned Building a Scrabble App for My Friends"},{"body":"This is the last post in the IZE series. In the previous installment, I looked at two ways to generalize the IZE algorithm itself: preferring consistent facets and searching for trees with better goodness scores. Here I want to ask a different question: what does the AI revolution of the last few years actually change about what could be built here?\nI see three distinct opportunities, at different layers of the system.\nThree layers where AI supports or leverages IZE-style hierarchical search Weighting splits for semantic search The IZE algorithm's core operation is: find the tag that, if used as a split, puts the most items in the \u0026quot;yes\u0026quot; branch. It's a raw count. That works well for keyword or boolean retrieval, where all results are retrieved by a hard match and any notion of \u0026quot;how well does this item match the query?\u0026quot; is addressed by ranking.\nSemantic (vector) search changes that. Instead of a binary retrieved/not-retrieved, each result comes with a similarity score. A document that closely matches the query vector scores higher than one that's only vaguely related. Retrieval becomes a numeric threshold operation that happens after ranking, instead of before. The current IZE algorithm ignores the ranking signal entirely.\nA natural fix would be to weight each item's contribution to the split by its similarity score. Instead of counting how many items have a given tag, you'd sum the similarity scores for items with that tag. The algorithm picks the split that maximizes this weighted sum. The effect is that splits capturing the most query-relevant items are preferred, not just the largest groups. A tag shared by five highly-relevant items would beat a tag shared by eight weakly-relevant ones.\nThis change is modest in implementation -- a drop-in replacement for the count function -- but its behavioral consequences are meaningful. With vector search, there's no hard threshold for what counts as a \u0026quot;result\u0026quot;; you're always choosing a cutoff somewhere. Weighted splits let you defer that decision, naturally favoring splits that correspond to concentrations of relevance in the result set.\nLLM-generated tags and labels Traditional IZE labels are just the keyword itself. The algorithm splits on \u0026quot;brand:Nike\u0026quot; and displays either that or \u0026quot;Nike.\u0026quot; For well-structured catalog metadata this is fine, but it's limiting in two ways.\nThe first limitation is on the input side. If your documents are free text -- product descriptions, internal wiki pages, email threads -- you either need users to manually tag everything (the original IZE burden) or run some kind of extraction. LLMs are genuinely good at this, though it's worth noting that older ML approaches (named entity recognition, text classification) worked reasonably well on structured domains like product catalogs. LLMs are just much more general: they can extract tags from arbitrary free text without domain-specific training data. Given a product description, a model can extract structured attributes -- material, use case, price tier -- at index time, and the resulting tags feed directly into the IZE algorithm without changing it.\nThe second limitation is on the output side, and it's more interesting. A node split on \u0026quot;outdoor AND lightweight AND under $50\u0026quot; gets labeled in the current system as those three facet labels, on separate rows. That's accurate but not particularly readable, especially as the hierarchy deepens. An LLM could generate a short natural-language label for each node -- \u0026quot;budget hiking gear\u0026quot; or \u0026quot;entry-level camping essentials\u0026quot; -- using the facet values and a sample of the items in that node as input. This label could be more useful to a browsing user than the raw conjunction of filters.\nThe challenge might be latency and coverage -- computing these labels in real time might slow down user interactions. And you can't pre-compute labels for every possible query result. But you can pre-compute the common cases, cache new labels as they arise, and fall back to facet-value labels for the long tail.\nIZE hierarchies for agentic orientation This one is more speculative. AI agents doing retrieval -- browsing documents, fetching context, answering questions over large corpora -- have an orientation problem that's structurally similar to the one IZE was designed to solve for humans. An agent dropped into an unfamiliar knowledge base doesn't know the vocabulary, doesn't know the distribution of content, and has to issue multiple queries to figure out what's there. Each query costs tokens and latency. The agent might retrieve irrelevant documents, miss relevant ones, or loop through the same territory repeatedly.\nAn IZE hierarchy over the corpus is a compact representation of its structure and contents. A one-page hierarchy for a moderately complex corpus might fit in 50-100 tokens and give the agent a useful map before it issues a single retrieval query. The agent could look at the top two levels, decide which branch is relevant to its current task, then drill into that subtree -- issuing targeted queries rather than broad exploratory ones. This is token-efficient in a way that sending a large retrieved set to the model isn't.\nThe hierarchy also might make agent behavior more predictable. Rather than issuing open-ended queries and hoping for good retrieval, the agent follows a structured path. That structure can be logged, audited, and optimized. Whether this would actually improve agent performance in practice is an open empirical question, but the mechanism seems sound. The same intuition behind IZE's user interface design -- \u0026quot;a search UI should educate as well as retrieve\u0026quot; -- applies to AI systems navigating unfamiliar document spaces, not just to humans.\nThat's the end of the series. To recap: IZE was a 1988 DOS application that built dynamically-generated hierarchies from tagged documents. It was commercially unsuccessful but technically interesting, and its core idea -- splitting search results recursively on the most informative single keyword -- was re-invented and studied independently in several different domains. Today, with rich catalog metadata, vastly faster computers and algorithms, and LLMs, the algorithm is more tractable than ever. E-commerce and enterprise search use-cases seem relevant, and modern AI technology could be both a user and a driver of effective systems.\nWhether any of this gets built in a product is a different question. If you have a compelling use case, please reach out.\nNotes:\nThis post was primarily human-authored, with AI assistance for research, editing, diagram drafting, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine. ","date":1773792e3,"id":"8","link":"/2026/03/ize-meets-ai-semantic-agentic-search/","section":"post","summary":"This is the last post in the IZE series. In the previous installment, I looked at two ways to generalize the IZE algorithm itself: preferring consistent facets and searching for trees with better …","tags":["search","history","ize","ai","llms"],"title":"IZE meets AI -- semantic search, smarter labels, and agentic orientation"},{"body":"In previous posts, I showed off an interactive demo of the IZE algorithm, and discussed how the algorithm worked. Now, it's worth considering some ways we could generalIZE the algorithm. 🤦‍♂️ Perhaps variations on the algorithm might yield hierarchies that are even better at showcasing the contents of the texts?\nAs discussed previously, there's pretty good experimental evidence that in order to be useful and intuitive, search results clustering needs to be based on just one or two words (or facet values, or tags). That is, it's too confusing when the algorithm finds clusters based on overall family resemblance, then tries to name them post-hoc. It's better to do the sort of single-word splitting that the IZE algorithm uses. But, could we do so in a different way or a different order, so that the results feel more like underlying clusters are being identified?\nConsider an e-commerce application. If your catalog was just Clothing and Accessories, you would want to ensure that those were top-level splits, even if they didn't happen to be the most common facet values. The existing IZE algorithm is greedy -- at each step, it finds the single tag split that maximizes the number of items in the Yes branch. This is very fast (assuming you can quickly calculate the number of items that result from each split). But it may not be optimal, for some particular definition of optimal. To explain, let's review the way the algorithm works.\n(A) Underlying binary tree; (B) IZE hierarchy as displayed The recursive splitting algorithm finds the next tag that maximizes the number of items in the next Yes (right) branch. The display converts the underlying tree into an easy-to-read rotated hierarchy. There is one row per leaf node (grey circles, with sets of results), plus one row for completely internal nodes (here, \u0026quot;clothing\u0026quot;). Note that the \u0026quot;footwear\u0026quot; and \u0026quot;sneakers\u0026quot; nodes are combined because they are redundant.\nPrefer consistent facets One simple enhancement is relevant to the e-commerce use-case, where tags are part of facets. The existing algorithm treats all tags similarly, but it would be straightforward to prefer certain tags (facet values) in certain circumstances. For example, when splitting a \u0026quot;no\u0026quot; node, you could prefer facet values that are in the same facet as the parent. The results are more intuitive. The sequence of 'no' nodes that make up the left edge of the tree -- rendered at the same level of the IZE hierarchy -- are more likely to share a facet.\nStandard IZE algorithm (left) vs. same-facet-boosted algorithm (right) showing brands consistently at the top level. Optimize for similarity Even more ambitiously, can we find a way to search through a larger set of possible trees so that the resulting IZE hierarchy is more similar to the underlying clusters?\nIt's helpful to think of a space of possible trees, where each tree T defines a series of splits that yields an IZE hierarchy. There could easily be billions of possible trees, depending on the number of tags and items, each defining a different IZE hierarchy for the same search results. Each tree T could then have a \u0026quot;goodness\u0026quot; score, G(T). There are many ways to define this score, but a reasonable one is to maximize how similar the items are within each node, and to minimize how similar the items in separate nodes are from each other. For e-commerce items, which typically have a set of tags (facet values), one option is to compute the Jaccard similarity between items. G(T) then becomes the sum of the Jaccard similarities between all pairs of items in each node, minus the sum of the Jaccard similarities between all pairs of items in different nodes.\nSo, could we enumerate all possible trees, compute the G(T) values, and pick the best? Well, no, not at any scale. Even with just dozens of tags, and hundreds of items, the computational complexity makes a real-time search system infeasible. Some sort of approximate or heuristic solution will be needed. We just need good-enough performance in bounded time (say, 1 second) for realistic use. A number of optimizations would need to be used:\nInstead of enumerating all possible trees, use a heuristic search approach to limit the search to trees that are most likely to yield good results. (AI techniques from the 20th century!) Store lots of intermediate states. Many trees differ by just a subset of their nodes, so only the differences need to be compared. Everything else can be computed once and looked up. Identify equivalent trees. There are many re-orderings of splits that yield the same nodes, just in a different order. We want to prefer the order where there are more items at the top of the list (nodding to the existing heuristic), and the other re-orderings can be pruned and ignored. Implement inside the search system. Modern search systems have highly-optimized representations of subsets of items that can be used for these algorithms. Certain invariant properties of items and facet values can be pre-computed, as well. Stay tuned for the final episode of this series -- thoughts on how the AI revolution of the last few years affects what could be built in this space.\nNotes:\nInterested in implementing this with me? Have a compelling commercial use-case? Please reach out! This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine. ","date":1773705600,"id":"9","link":"/2026/03/generalize-ize-hierarchies/","section":"post","summary":"In previous posts, I showed off an interactive demo of the IZE algorithm, and discussed how the algorithm worked. Now, it's worth considering some ways we could generalIZE the algorithm. 🤦‍♂️ Perhaps …","tags":["search","ize","algorithms"],"title":"GeneralIZE -- How else could IZE's hierarchies be generated?"},{"body":"In the previous installment of this series, I looked at what came after IZE -- faceted search, clustering algorithms, and the various ways web search, personal information management, and e-commerce tried to solve similar problems to what IZE was attacking. None of them ended up doing what IZE did. The question I want to take up here is: could we build something like IZE today? I think the answer is yes, for at least one domain. IZE's hierarchical search algorithm is worth revisiting for e-commerce operators with medium-to-large catalogs who want to help new customers orient themselves. Modern catalog metadata makes it easier to implement than ever.\nWhy e-commerce is the right fit What would a modern product leveraging the IZE algorithm look like? The original product required users to manually and consistently tag their documents with keywords, which was a significant burden. What if we thought of IZE as a search UI pattern, instead of a product? Modern e-commerce catalogs already have rich metadata for filtering and faceted display -- brand, category, color, size, material, price range, and so on -- that could serve as a replacement for those user-defined keywords. You don't need to ask anybody to tag anything new, and you don't need to worry about stopwords or morphology issues that arise in free-text search. The facet values are already there.\nCatalogs tend to have hundreds up to millions of items, and users often don't know what they're looking for. Medium-to-large catalogs are exactly where new-user disorientation is most costly.\nAs I noted in the first post in this series, a search UI has two jobs: helping people find relevant information, and educating them about the structure of the information space. IZE was unusually good at the second job. Most modern search UIs focus on the first job, and do a weaker job at the second.\nFaceted search is fine if you know you want \u0026quot;blue sneakers, size 10, under $100.\u0026quot; It's less great if you're browsing a new-to-you outdoor gear store and you're not sure what category your need falls into. A dynamically-generated hierarchy could orient you to the available content in a way that faceted search doesn't. This is the specific gap IZE was designed to fill: not just finding, but educating users about the shape of the result set.\nHow the algorithm works What I find appealing about IZE's approach is that it's synthetic and dynamic. In the original implementation, as described in a previous post, the hierarchy isn't pre-defined by a taxonomy team; it emerges from the documents themselves, and their user-specified keywords, in response to the user's current query. That's different from faceted search, where the facets are orthogonal and fixed in advance.\nWhen applied to an e-commerce catalog, the algorithm works just like the original IZE algorithm, but using the catalog's facets instead of user-defined keywords. After the initial search, which provides the result grid, the algorithm makes subsequent queries, leveraging a faceted-search feature that provides counts of items that match each facet value. For example, if the user searches for \u0026quot;blue phone\u0026quot;, the search engine might indicate that \u0026quot;brand:Samsung\u0026quot; has 100 matches, \u0026quot;brand:Apple\u0026quot; has 50, and so on. The algorithm uses these counts to greedily select the splits, then recurses to build the IZE hierarchy. The result is a dynamically-generated hierarchy of facet values, which the user can navigate to understand and explore the catalog. Unlike the original IZE, we'd implement a two-column UI: the hierarchy on the left, the filtered items on the right, updating as the user navigates.\nThere's also a relationship here to post-query refinement suggestions, which I wrote about earlier. PQRSs are a lighter-weight version of the same idea -- after you submit a query, the system suggests filters that would help you narrow the results. IZE's approach is more structured and recursive, but the intuition is similar: show the user the shape and scope of the result set, not just the results.\nA demo To explore these ideas, I built a naive implementation on top of an existing search engine with demo data.\nIZE demo app with no search terms The implementation is not difficult, but it has a significant performance problem: the recursion requires re-querying the search engine for each node. So, rather than searching for \u0026quot;blue phone\u0026quot; once, behind the scenes you might be searching for \u0026quot;blue phone\u0026quot; and \u0026quot;blue phone AND color:blue\u0026quot; and \u0026quot;blue phone AND color:blue AND brand:Apple\u0026quot; and so on. This is not a production-ready approach, but it does show what the two-column UI might look and feel like.\nWant to try it out? The demo is here. It's a bit buggy, but it's fun to play with! You can compare the standard faceted search UI with what the IZE algorithm produces, and note how much more you learn about the structure of your search results from the IZE algorithm.\nComparison of IZE hierarchy with faceted search The IZE hierarchy is more intuitive and easier to navigate, it does not require scrolling, and it provides a better sense of the structure of the result set. However, for experienced users, it may be less efficient to filter on a specific, known facet value.\nCould it be optimized? The performance of the algorithm could be improved, with some engineering effort. The recursion is expensive because we're starting from scratch with a new query at each step, and because the query itself does ranking and many other features that aren't needed for the hierarchical clustering. An implementation using algorithms that re-use intermediate steps, such as the roaring bitmaps generated for specific sub-searches, could be much faster -- fast enough, I think, to be viable for production use on medium-to-large catalogs. The bitmaps could be cached and re-used for subsequent sub-queries, and the recursive algorithm could be optimized to avoid redundant or unnecessary work.\nThe product question The real barrier isn't engineering -- it's operator adoption risk. Users are trained on faceted search; a new pattern requires a reason to switch. The case for trying: new-user conversion on large catalogs is a real, measurable problem, and the IZE approach is a testable hypothesis. An A/B test comparing standard faceted search against a dynamically-generated hierarchy for new users on a large catalog would be a reasonable way to find out.\nIn the last two posts of this series, I'll be covering extensions and variations on IZE that might yield even more compelling and intuitive hierarchies, and will explore the implications of modern AI on IZE-based search UI.\nAnd if you're interested in implementing some variation of the IZE algorithm efficiently, for a real business problem, I'd love to hear from you.\nNote: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1773014400,"id":"10","link":"/2026/03/could-we-build-ize-again/","section":"post","summary":"In the previous installment of this series, I looked at what came after IZE -- faceted search, clustering algorithms, and the various ways web search, personal information management, and e-commerce …","tags":["search","history","ize"],"title":"Could We Build IZE Again?"},{"body":"In the previous post in this series, I discussed the technical details of IZE and its reception. Here I want to look at what came after — and where IZE-like ideas might still have potential.\nThe short version: IZE was forgotten, but the ideas it embodied — hierarchical clustering, single-word splits, dynamic navigation — were re-invented independently in several different domains. Each domain found a different answer, for reasons that are worth understanding.\nThe Open Web: Keyword Search and Ranking Win IZE was introduced in 1988. The first graphical web browser, NCSA Mosaic, was released in 1993. Within a few years, hundreds of millions of people were using the web, and the need for better search and navigation tools became apparent.\nThe initial solution was a category structure. Web sites were manually curated into hierarchical directories for discovery — Yahoo! being the canonical example. As Hearst (2009) notes:\nA fixed category structure helps define the information space, organizing information into a familiar structure for those who know the field, and providing a novice with scaffolding to help begin to understand the domain. (Hearst, 2009)\nThis worked until it didn't. The web grew faster than any editorial team could curate it, and hierarchical directories collapsed under scale. Engines like Google (1996) took a different approach: don't organize, rank. Powerful keyword-based retrieval combined with algorithmic ranking (PageRank) made web site categorization largely unnecessary. The search UI eventually became \u0026quot;10 blue links\u0026quot; with pagination and relatively minimal use of filters.\nIn academia, meanwhile, clustering approaches tried to impose post-hoc hierarchy on web results. The Scatter/Gather algorithm (Cutting et al., 1992) is an example. It grouped similar documents together and let users navigate the resulting clusters. Each cluster was defined by a set of keywords, and documents were assigned based on their similarity to those keywords. Compared with IZE, which used a single keyword to split results at each stage of a hierarchy, Scatter/Gather and other algorithms in this class used multiple, often hundreds of keywords per cluster. The clusters may have been coherent, but the lists of words defining them were not intuitive to users. Hearst's verdict is blunt: \u0026quot;The disadvantages of [many-word] clusters for user interfaces include their lack of predictability, their conflation of many dimensions simultaneously, the difficulty of labeling the groups, and the counter-intuitiveness of cluster subhierarchies.\u0026quot;\nThe approach most similar to IZE from this era was Findex (Kummamuru et al., 2004) — almost certainly an independent re-invention, since IZE was not widely cited by this point. Findex used a hierarchical clustering algorithm to group web search results into a tree, then let users navigate it. Unlike Scatter/Gather, Findex used a single word or phrase to split results at each level — much closer to IZE's design. It used document titles and search snippets to identify frequent keywords, automatically handling stopwords and morphology. User studies showed it was most helpful with broad or ambiguous queries, or in early exploration of unfamiliar document sets.\nFindex clustered search results, from Hearst (2009) Clusty.com was a commercial attempt at this -- clustering plus incremental refinement, applied to web search results.\nAs users entered queries — for example, \u0026quot;Walt Disney\u0026quot; — Clusty automatically created categories such as \u0026quot;Walt Disney World,\u0026quot; \u0026quot;Collectables,\u0026quot; \u0026quot;History,\u0026quot; and \u0026quot;Biography\u0026quot;. This categorized view allowed users to focus on the subtopic most relevant to them, reducing the need to sift through irrelevant results. (From an article about Clusty on Yippy.com)\nYou probably haven't used Clusty.com in a while. It eventually became clear that clustering was not a good fit for web search at scale:\nIt appears that, although hierarchical structures are commonly used and are useful when applied to smaller collections, as a navigation structure, they become unwieldy when applied to very large collections. (Hearst, 2009)\nThis domain is, I think, settled. Dynamic hierarchical clustering lost on the open web. It just doesn't work at that scale, with such heterogeneous content, and such a diverse user base.\nPersonal Information Management (and Enterprise Search): Still Contested The most successful personal information management tools -- Evernote, Obsidian, Tana, Roam -- let users create, search, and navigate their own documents. They rely on a combination of full-text search (Evernote excels here), manual or semi-automated tagging, and graph views (Obsidian). A graph view lets users see relationships between documents and navigate to related ones.\nThese tools are not based on clustering or faceted navigation. They're bets on different models of how people think about their own knowledge: as a flat searchable archive, as a network of linked ideas, as an outline. Each has devoted users. None has clearly won.\nEnterprise search is the large-scale cousin of this problem: same challenge, higher stakes, often worse tools. A new employee trying to find relevant internal documents faces the same orientation problem as a user with a large personal archive — they don't know the vocabulary, they don't know what's there, keyword search only helps once you know what to search for. Plus, with many users, sources are wildly different, content gets stale or is conflicting, and hand-curating a table of contents typically fails.\nAn IZE-like approach — dynamically clustering your documents as you accumulate them, surfacing a navigable hierarchy without requiring you to tag everything manually — doesn't seem to have been seriously tried here. I'm not sure it would fail. The collection is smaller than the web, the user is often a domain expert, and orientation matters a lot. Whether algorithmically-generated hierarchies would help, or whether the lack of user control would just feel alienating, is genuinely unclear to me.\nE-Commerce: Faceted Search Has Won, But the Job Isn't Done E-commerce went a different direction from web search. Because sites control their own catalogs, they can manually or automatically add consistent metadata to their products. In the early 2000s, academic projects such as Flamenco (Hearst, 2000; Yee et al., 2003) investigated faceted navigation, where each item has multiple orthogonal categories and users can filter by any of them. This work was quickly adopted by Amazon, eBay, and others. Today, faceted search and navigation is a standard feature of most e-commerce sites.\nFlamenco faceted search, from Hearst (2009) Facets work because the domain structure is pre-existing and stable. The catalog owner controls the vocabulary; the facets can be designed to match how users think about the product space. As Hearst points out, \u0026quot;If the facets do not reflect a user's mental model of the space, or if items are not assigned facet labels appropriately, the interface will suffer some of the same problems as directory structures.\u0026quot; A mismatch signals to users that the site doesn't understand them, and they'll trust it less.\nBut faceted search has a known failure mode: users who don't know the vocabulary can't filter effectively. Someone browsing a new-to-them catalog — unfamiliar with the product categories, the brand names, the relevant attributes — often can't use facets productively. They bounce, or fall back to keyword search and struggle. The orientation problem is still unsolved.\nCould a dynamic clustering layer above facets help users orient before they filter? That's the question I'll be taking up next. AI changes the picture too — LLM-powered query understanding, conversational search, and semantic retrieval all attack the orientation problem from different angles, and I'll get to those as well.\nThe idea of algorithmically generating a navigable hierarchy from a document collection didn't disappear with IZE — it was re-invented, independently, in each domain. On the open web, it lost to ranking. In personal information management and enterprise search, it was never really tried. In e-commerce, faceted navigation won the orientation battle, though not completely. The places where IZE-like ideas still seem interesting are exactly the ones where users arrive without a clear vocabulary: a new enterprise knowledge base, an unfamiliar product catalog, a personal archive they haven't touched in years.\nNotes: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1772582400,"id":"11","link":"/2026/03/what-came-after-ize-three-domains-three-answers/","section":"post","summary":"In the previous post in this series, I discussed the technical details of IZE and its reception. Here I want to look at what came after — and where IZE-like ideas might still have potential.\nThe short …","tags":["search","history","ize"],"title":"What Came After IZE? Three Domains, Three Answers"},{"body":"In the first post in this series, I introduced IZE -- a DOS-era personal information manager with a novel approach to search and navigation. Here I want to go deeper into how it actually worked, what its limits were, and how it was received at the time.\nThe algorithm The core of IZE was patented by Paul Kleinberger (US5062074A, \u0026quot;Information retrieval system and method\u0026quot;). The basic algorithm is easy enough to describe:\nFirst, filter the documents to include only those that contain the search term. Then, for each document in the results set, count the number of times each keyword appears. Find the most frequently occurring keyword across the result set, and split the results into two groups -- those that include that top keyword, and those that don't. This defines the root node of a hierarchy tree. Then, recursively apply the same process (return to step 2) to each group, splitting the largest nodes first, using the top keyword in each group as the next split. Stop when there aren't any documents left to split, or when you've filled up the screen (about 15 rows). Consider this example:\nIZE screenshot After the first step, finding documents with \u0026quot;new\u0026quot; tagged in the Letters textbase, there are 14 documents remaining. The most frequently occurring keyword in those documents is \u0026quot;announcements\u0026quot;, which appears 5 times. So the first split is into \u0026quot;announcements\u0026quot; (5 documents) and \u0026quot;not announcements\u0026quot; (9 documents). The next recursive step would be to break down \u0026quot;not announcements\u0026quot;, which would split into \u0026quot;welcome\u0026quot; (4) and \u0026quot;not welcome\u0026quot; (5). Note that \u0026quot;not announcements\u0026quot; and \u0026quot;not welcome\u0026quot; are not explicitly shown in the UI, but they're present in the underlying tree structure.\nflowchart LR A[Announcements] --\u003e|Yes, 5| B[to be determined next] A --\u003e|No, 9| C[Welcome, 4] The algorithm is simple, but it's also surprisingly effective. It's a form of divisive hierarchical clustering, where the splits are determined by the most frequent keyword in the current set of documents.\nThe binary tree that results gets converted into an indented hierarchy, which makes it easy to display and navigate. Selecting a subtree drills down, re-building the hierarchy from the subset of documents. Or you could go to a specific document by selecting the ▶️ symbol. There's some additional complexity related to fully redundant keywords, and some details related to how the hierarchy is displayed on a 24-row screen, but that's the core of it.\nIt's worth comparing IZE's approach with contemporary search engines. They also filter to include items that match the search term -- that's the \u0026quot;retrieval\u0026quot; step. But they also rank results to show the most relevant documents first. IZE didn't rank results at all. Instead, it gave you structure -- a way to see what kinds of documents matched your query, and to navigate to the ones you cared about. The tradeoff was that you couldn't just scan the top few results; you had to engage with the hierarchy. But for some use cases that was arguably the right approach.\n1980s search engine IZE (1988) Modern search engine Search Boolean (AND/OR/NOT) Boolean Keyword + semantic Ranking Yes, by relevance score None Yes, by relevance + personalization Orientation Flat result list; you're on your own Keyword-split hierarchy; browse the structure Faceted filters, query suggestions, related searches Topical search One feature worth calling out was what IZE called \u0026quot;topical search.\u0026quot; When you drilled down through the hierarchy, you were following a strict top-down path -- each level narrowed the set further. But topical search let you reset the search from your current context, which could include items that would otherwise have been excluded by the path you'd taken.\nIn the running example above, you could do a topical search for \u0026quot;welcome\u0026quot;, and it would show all results with \u0026quot;new and welcome\u0026quot;, even if they did have \u0026quot;announcement\u0026quot; in them as well. This was a mechanism to let you browse the structure more freely, and supported a more exploratory workflow.\nLimits IZE had some real constraints. The initial search was optionally full-text, but refinement was only based on the pre-defined keywords used to define the hierarchy. Support for natural language queries was limited -- there was only basic handling of affixes and synonyms. The system had a hard limit of 32,000 documents, whether internal or external (linked but not imported). Keywords could be selected manually in the text editor, or you could define rules to auto-tag documents -- something like \u0026quot;add this keyword if it's present\u0026quot; -- but there was no sophisticated natural language processing going on. This was the late 1980s and inexpensive PCs, after all.\nEsther Dyson's take Esther Dyson covered the technology that would become IZE (then known as TNET) in her influential newsletter Release 1.0 in May 1987. For those unfamiliar, Dyson was one of the most respected technology analysts and investors of the era -- her newsletter was required reading for anyone in the PC software industry. Getting a writeup in Release 1.0 was a significant endorsement.\nFrom a writeup in Release 1.0 She then opines:\nThe beauty of TNET is that it does the work. Automatic indexing is easy; automatic structuring based on such an index, to our knowledge, isn't commercially available. It can be scaled up; indeed, its value increases on large-scale text bases (although performance may be an issue)... Unlike typical text search programs, where you can get exactly what you ask for whether or not you ask for the right thing, TNET helps you explore what you could want -- not in the usual way of providing a list of key words, a ranking of items by number of matches with a list or cluster of key words, but in a meaningful way, so you can easily see what you have to choose from.\nProduct history IZE was acquired and then published by by Persoft, a Madison, Wisconsin-based software company. After Persoft was acquired a few years later, IZE was spun off to a new company called Retrieval Dynamics, Inc. IZE's sales weren't strong enough to sustain the company, and it was discontinued.\nNow, you can download IZE from abandonware site Vetusware, and run it in a DOS emulator. I wouldn't necessarily recommend you do so, but as I'll discuss in future posts, it may provide some inspiration. But first, I'll look at technologies that came after IZE -- faceted search, outliners, topic models -- and how the problem IZE was trying to solve has been addressed (or not) by later systems.\nNote: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1772236800,"id":"12","link":"/2026/02/how-ize-really-worked-patents-limits-esther-dyson/","section":"post","summary":"In the first post in this series, I introduced IZE -- a DOS-era personal information manager with a novel approach to search and navigation. Here I want to go deeper into how it actually worked, what …","tags":["search","history","ize"],"title":"How IZE Really Worked - Algorithm, Patent, Limits, and Esther Dyson"},{"body":"Sometimes revisiting old technology is the best way to understand how we got where we are -- and to see what alternative paths might have looked like. This is the first in a series of posts about IZE, a DOS-era personal information manager that I think has some interesting lessons for modern search and discovery.\nI'll admit my interest in IZE isn't purely academic. My father was a co-founder of the company that published it, and a cousin of his was the inventor of the underlying technology, which the company acquired. So I grew up with IZE around the house, and owned the polo shirt. That said, I think it's genuinely interesting on its own merits, and I'll try to make the case for that here.\nWhat was IZE? IZE splash screen IZE was a DOS application introduced in the late 1980s. It was, broadly speaking, a personal information manager -- a word processor with organizational and information retrieval features layered on top. Think of it as a precursor to tools like Obsidian, but running on a 286 with a monochrome monitor and no internet connection.\nThe innovation that set IZE apart was its approach to search and navigation: an interactive, dynamically-generated hierarchy. When you searched your document collection, IZE didn't just return a flat list of matching results. It built a table of contents on the fly, organized by the most frequently co-occurring keywords in your result set. Drill into a branch of that hierarchy, and IZE regenerated it based on the narrowed set of documents. The keywords could be selected manually or semi-automatically, depending on how much you wanted to guide the process.\nIt was a novel attempt to give users context for navigating a text collection and refining their queries -- years before faceted search became a subject of serious academic study, and two decades before it became the default pattern for e-commerce and enterprise search. A Search UI has two jobs -- helping people find relevant information, and educating them about the structure of the information space. IZE was a surprisingly good for the era attempt at the latter.\nHow good? Although its sales were not strong enough to sustain the company, one enthusiastic user got an IZE logo tattoo!\nWhat it looked like IZE screenshot The interface was text-mode, as you'd expect for a late-80s DOS application. The hierarchy appeared as an indented outline, and a well-designed set of keyboard shortcuts made it easy to navigate to find the most relevant documents. It was surprisingly fluid for the era.\nIn the screenshot above, from a tutorial textbase of business letters, you can see that I'd searched for \u0026quot;new\u0026quot;, and IZE generated a hierarchy of documents that contained that word, organized by the most frequently co-occurring keywords. The cursor is on \u0026quot;new business announcement letters\u0026quot;. It's very fast to see what sorts of \u0026quot;new letters\u0026quot; are available, and to navigate to the ones you want.\nWhy this matters now The question of how best to help users orient themselves within a large document collection (or e-commerce catalog) -- and how to help them refine their queries -- is still a topic of research and experimentation. Faceted search, which became dominant in the 2000s and 2010s, addresses part of this. But facets require a pre-defined taxonomy, and they work best when the attribute space is relatively flat and well-understood by the users. IZE's approach was different: it derived structure from the keywords tagged in the documents themselves, dynamically, in response to the user's current query.\nI think it's worth understanding that idea carefully before asking whether modern tools -- vector search, topic models, LLM-generated labels -- could do something similar, or better.\nComing up in this series This post is just the introduction. Here's what I'm planning to cover:\nHow IZE really worked -- patents, limits, and Esther Dyson's take What came after IZE? Three Domains, Three Answers Could we build IZE again? A live demo GeneralIZE -- How else could IZE's hierarchies be generated? IZE meets AI -- semantic search, smarter labels, and agentic orientation I'll update this post with links to the follow-ups as they're published.\nNotes: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine. Thanks to Ed Harris for feedback and valuable context.\n","date":1772150400,"id":"13","link":"/2026/02/ize-revisiting-hierarchical-search-technology-pc-era/","section":"post","summary":"Sometimes revisiting old technology is the best way to understand how we got where we are -- and to see what alternative paths might have looked like. This is the first in a series of posts about IZE, …","tags":["search","history","ize"],"title":"IZE - Revisiting a hierarchical search technology from the PC era"},{"body":"I've supported A/B testing of Algolia search systems at three companies now, and have dived deep into A/B testing generally as well as specifically for search. The Algolia documentation on search A/B testing is technically adequate for getting started, and the dashboard has improved, but there are still many ways that you can go wrong when A/B testing Algolia search results. In the style of a 2014-era Buzzfeed listicle, here are 11 Algolia A/B Testing Gotchas, Tips, and Lessons!! All of the horrible illustrations are generated by AI, the rest are from earlier posts.\nBefore you start the test Replicas must be identical except for what you're testing. Although some parameter settings can be tested in real-time, most A/B tests will be on replicas with different ranking criteria, tiebreakers, or settings that require a separate index. The catch: Indexes get out of sync incredibly easily, invalidating your tests. If someone adds a synonym or rule to your control index, without replicating to your test index, any difference you see in metrics could be caused by that change, rather than by whatever you actually intended to test. This has happened to me many times. My recommendations:\nUse the Algolia CLI to diff the indexes, before you start the test, and every couple of days thereafter. Proactively reach out to engineers and business people with Algolia access and let them know that seemingly innocuous changes to production configurations can delay work by weeks. Get used to frustration and re-starting tests. 2. A/B testing when you're using the dynamic reranker is fraught. The dynamic reranking feature is really valuable -- it uses user interaction patterns from events to automatically improve ranking for the most common queries. The impact on relevance can be substantial. But the feature doesn't play well with A/B testing.\nPlot twists: When you set up a replica index, dynamic reranking is by default off, even if the original index used that feature. Then, the standard way of enabling dynamic reranking looks at interactions with that index, which for a test index is likely starting from scratch! So you're testing a control with a well-tuned reranker against a test index with no or poor reranking -- it'll never win. There is a way to work around this -- the test index can use the control index as the source of events. Annoyingly, this is mentioned on the reranker docs, but not the A/B testing docs.\n3. You need to QA before starting the test. As I've written previously, Algolia's otherwise solid Dashboard lacks a good tool for doing head-to-head tests. Fortunately, it's not that hard to build one yourself. Pro tip: Before starting a test, you and stakeholders should use a tool like this to make sure you understand the implications of whatever you're changing.\nOffline analysis can support QA. In the past, I've written scripts that do the following: Pull a representative sample of hundreds of historical user queries from the data warehouse. Use the Algolia API to pull search results for those queries, on both the control and test indexes. (Remember to set analytics:false to avoid corrupting your own data and AI features!) Build dashboards (I used flexdashboard at the time, but now you could vibe-code anything) that let you see which queries had big changes to the results, if the changes affect metrics such as \u0026quot;average price on page 1\u0026quot;, \u0026quot;average popularity of top results\u0026quot;, etc. Dive into the results and use your head-to-head tool to deeply understand changes before going live. 5. Algolia only supports full-population A/B tests. Sometimes certain users shouldn't be included in an A/B test or an A/B test analysis. On a two-sided marketplace, you may want to exclude sellers from a test, to minimize confusion before a feature is released. Or you may only want to test users coming from a specific marketing source. Nope, Algolia's built-in A/B testing can't do this. For anything fancier than testing everybody, you'll need to use separate A/B testing tools to handle bucket-assignment. See also #9 and #10 below.\nAnalyzing A/B test results 6. Algolia's A/B test analysis only works at the query grain. In A/B test analysis, grain is the unit of a test. For search, the grain can be a query (\u0026quot;blue shoes\u0026quot;), a subquery (\u0026quot;blue sho\u0026quot;, if you have search-as-you-type in your UI), a search session (multiple searches for a single intent), or even a user funnel (from the first time you see a user, until they make a purchase, if they do). Algolia's A/B test analysis only works at the query grain. That may not be what you want.\nFor many e-commerce use cases, users refine their initially-broad queries, as they understand better the scope of your catalog. This is good! And your marketing team has a carefully designed abandoned-cart campaign to get users back on the site days later. An A/B test analysis that penalizes poor conversion rates from initial, broad searches like \u0026quot;shoes\u0026quot; may well not be serving your users (or your business) well. Ideally, look at search-session or even multi-session funnels, to determine if a changed search experience increases eventual purchase rates or revenue.\nIf you're relying on Algolia's A/B test analysis alone, you need to understand the limitations and use the results you get as a proxy for the data you actually want.\n7. You can do your own analysis. Good news: When a query is part of an A/B test, the Algolia API will return the abTestID and abTestVariantID in the results object. If you can capture this and send it to your event-tracking system, along with user ID information, you or your data scientists can use your analytics tool or data warehouse to analyze the results. This has many advantages -- you can measure metrics that you aren't sending to Algolia (are users rating their purchases higher?), you can fix the grain issue and look at search sessions or user funnels, you can more carefully exclude internal users and bots, and you can look at subgroups of users separately (do new users behave differently because of the change from returning users?). There’s some setup involved, but this is my recommendation. Aside from very lightweight tests, spend the time to set up the analyses the right way, for your domain and data.\n8. Algolia assumes you're making a Superiority decision. In earlier work, I described how many A/B tests are intrinsically lower-stakes, and being statistically confident that your change is better than the status quo actually is too high a bar. Instead, when making small changes, such as tuning search parameters or tiebreaker scores, you're actually making an Agnostic decision. As long as most of the time you choose the winning setting, the statistical test at the end isn't that important. Just choose whichever variant has even slightly better results.\nThe statistical method for being comfortable with this requires some understanding of how big the likely changes to your metrics will be, and some pre-test calculations to ensure you're getting enough data. You still need to collect data for a while, but you can often make a decision and move on even with traditionally-inconclusive results. Just don't quote the magnitude of the results you saw -- those numbers are close to meaningless from underpowered tests.\n9. Including all users can weaken effects. Suppose you're making a change that will only be experienced by a small number of users. Maybe your Test adds a synonym for \u0026quot;shooz\u0026quot; and \u0026quot;shoes\u0026quot;, I dunno. Algolia doesn't know which users are affected by this change, and treats all users as part of the experiment. But only users who typed either \u0026quot;shooz\u0026quot; or \u0026quot;shoes\u0026quot; could ever have seen anything different. So you're diluting the data you care about with a huge number of cases where any difference is just random noise. If you could only look at the users you cared about, you might see a 10% vs 20% difference -- huge! But when diluted by other users, it might only be 10.00% vs 10.05% -- minuscule, hard to detect, and hard to reason about.\nIf you do your own analysis (see #7), you can work around this. Define a funnel of users who could have been affected by the test, then compare conversion rates for those users. The numbers will be smaller, but the signal-to-noise ratio will be much better.\nOther notes 10. You don't have to use Algolia's A/B testing framework to test Algolia. The biggest business impact I ever created (roughly a 1.5% conversion-rate relative increase) was due to a coordinated change of a front-end UI feature with a tiebreaker weighting score change. On their own, neither change would have had the same impact, but the synergistic effect changed user decision-making for the better. This test was not run using Algolia's A/B test feature, but instead was run with an external framework that handled user-bucketing, analytics, and the simple logic that determined which user queries saw results from which index (control or test). To make this work, all that's necessary is to have a few lines of code somewhere that says \u0026quot;users in Control use index A while users in Test use index B\u0026quot;. We leveraged Algolia's replica-index features without using the A/B test feature.\nBurying the lede -- this is now my recommendation: in most cases, don't use Algolia's A/B testing framework. Algolia’s framework can be useful in limited cases, but you’re usually better off, using an external A/B testing framework, ideally one that integrates well with your Analytics system (see #7). Use Algolia for setting up the indexes, ensuring that only the intended change is different (see #1), and using the API for QA (see #3 and #4).\nYou'll also be able to more easily handle sub-population tests and analysis (#5 and #9), as well as more complex experimental designs, such as tests that ramp up over time while properly maintaining user-bucket assignment.\n11. The A/B testing feature can save your bacon. Years ago, a mistake I made corrupted our production Algolia index. Rebuilding the index, with millions of items, would have taken hours. Fortunately, we had a backup copy of the index in the same Algolia app, created daily by a cron job. (If you're not using the CLI and cron to kick off a daily backup, do yourself a favor...) But a production redeploy to point at the backup, or even copying the backup back to production, would have taken time we didn't have. I was able to set up a quasi-A/B test in just seconds. It redirected 100% of traffic from the corrupted primary index to the backup index, buying us time to re-index and fix things the right way.\nNow go run better Algolia A/B tests.\nAdvertisement: Does your organization struggle to get the relevance, user experience, and business impact you need from Algolia? I'm a freelance consultant with years of Algolia experience who can help you get the most out of advanced tools and search algorithms. Get in touch!\nNote: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1770595200,"id":"14","link":"/2026/02/algolia-ab-testing-gotchas-tips-lessons/","section":"post","summary":"I've supported A/B testing of Algolia search systems at three companies now, and have dived deep into A/B testing generally as well as specifically for search. The Algolia documentation on search A/B …","tags":["algolia","ab-testing","search","listicle"],"title":"11 Algolia A/B Testing Gotchas, Tips, and Lessons!!"},{"body":"Engineering leadership knows the standard playbook for product teams: the Product Trio, the Spotify model, outcome-based roadmaps, and so on. I've seen teams adopt these practices and still struggle when they add responsibility for advanced algorithms -- search, recommendations, predictive modeling, or generative AI -- without changing how the team is led. The processes and role expectations that work for a typical product team break down when the domain is heavily algorithmic. This post pulls together what I've learned and what others have written about building teams that own these systems.\nThe Product Trio Meets Advanced Algorithms A Product Trio gives you a standard set of operational processes: how the team communicates, how decisions get made, and what each role is expected to contribute. Product managers synthesize user needs and business goals, designers own the voice of the user and the interface design, and the engineering lead ensures that the system can be built, scaled, and maintained. What happens when the domain a team is responsible for is highly algorithmic -- search, generative AI, or other advanced algorithms? The Trio is no longer sufficient.\nThis isn't a new observation. Teresa Torres writes that if you work on a data-intensive product, your trio might become a quad: you may want to invite your data analyst to participate in most key decisions. Kevin Holland argues that data product teams need a data lead, not just occasional support from analytics. In search specifically, Daniel Tunkelang has written about how product and engineering leads on search teams have particularly overlapping roles. And James Rubinstein has unpacked the search PM role and why it's often misunderstood. These authors are all pointing at the same thing: when the core of your team's responsibility is an algorithm, the standard trio doesn't map cleanly onto the work.\nWhat Does This Look Like in Practice? So what skills, exactly, are needed? And who represents the voice of the user?\nFor search and similar algorithm-driven features, the voice of the user doesn't come primarily from interviews or usability tests. It comes from diving into a highly complex and stochastic set of logs and data sources, and from thinking through how algorithms shape user behavior. You need someone who can study query logs, click-through rates, reformulation patterns, and engagement metrics, and connect that to algorithmic changes that might help. That person needs both data skills and a deep enough understanding of the algorithms to reason about cause and effect. Many product teams have business analysts who can pull data and build reports, but for advanced algorithm teams, deep quantitative skills and the tools needed to rapidly discover and iterate are more-or-less the whole game.\nReganti and Badam's recent O'Reilly piece makes a related point for generative AI: evals are not enough. You need a process of continuous improvement, a \u0026quot;flywheel,\u0026quot; to deal with the complexity and stochasticity of an advanced algorithm. You can't just run a few tests before launch and call it done. The same logic applies to search and ML systems. The work is inherently iterative and data-driven.\nCompared with classical product teams, the code written for advanced algorithm teams is much less likely to be user-facing production code. There's a much higher need for internal tools such as team-specific reporting systems, interactive analysis tools, and prototypes for exploration. In search, that might mean tools to test new algorithms or UI patterns, alternative administrative UIs for specific business or technical problems, or custom dashboards that surface metrics that matter for your ranking and relevance. In ML, it means tools for building, scaling, and monitoring predictive systems. Generic analytics platforms rarely give you the right abstractions -- every advanced-algorithm product is too different, too context-dependent.\nGiven the advances in easy-to-use development frameworks and AI-assisted coding, it becomes easier than ever for the same person who is diving into the data and algorithms to build their own tooling. The data and algorithm lead becomes less dependent on a separate tools team, or on their team's spread-thin engineers, and more able to close the loop from insight to prototype to iteration.\nWho Leads, and Who Doesn't You still need Product -- for synthesis, coordination with external teams, prioritization, and making sure the algorithm work connects to business outcomes. You still need Engineering management -- for supervision, career development, and technical direction. But maybe you don't need design in the same way? Or at least, design plays a different role. The data and algorithm expert (whether you call them a data scientist, search scientist, or something else) provides the voice of the user, via the data. I think it's important to have the same person looking at the data and being capable of thinking through complex algorithmic changes. Splitting those responsibilities across a designer and an analyst who can't reason about the algorithm tends to create handoffs and lost context.\nSo, for an advanced algorithm team, the Product Trio becomes Product, Engineering, and Data \u0026amp; Algorithm leads. Design, as traditionally understood, becomes a supporting role, pulled in as necessary rather than driving the experience.\nThe Gen AI Era How does this change when we add Generative AI into the mix? Two angles matter: how Gen AI affects software development, and how it affects teams that deploy Gen AI systems.\nOn the software development side, the conventional wisdom is that in the world of Gen AI, many more developers can be \u0026quot;10x\u0026quot; developers compared to a few years ago. You don't need as many developers, and the bottleneck is likely to be figuring out what to build rather than building it. Marty Cagan and the SVPG team have written about how people on product teams will focus more on discovery, letting AI agents take more responsibility for delivery.\nFor advanced-algorithm teams, you still need the data science capabilities and the deep understanding of algorithms in order to determine what to build. The need for deep expertise doesn't go away, even if AI tools speed up aspects of data analysis, brainstorming, and more. Teams focused on Search or other advanced algorithms will get smaller and faster.\nWhen it comes to deploying Generative AI systems specifically, even though integrating an LLM is just a few lines of code, the implications for software lifecycle management are similar to other advanced algorithm systems, particularly Search systems. Teams building Gen AI products may find they have more in common with Search teams than with teams that ship traditional ML models. Gen AI systems, like Search, take free-form linguistic input and produce complex output, so evaluation, monitoring, and iteration look different from predicting a likelihood or a forecast from structured, tabular data. Even though the algorithms used in Generative AI and predictive modeling may be similar (deep learning), the product implications are very different.\nUI/UX considerations for Search and particularly chat-based Generative AI have much in common, as well. In both cases, the UI needs to help users understand what's possible, to meet them where they are, and to handle extreme variations in use cases and expertise.\nAll of which brings us back to team structure. If Gen AI products share more organizational commonalities with Search than with traditional ML, then the same team model applies: Product, Engineering, and Data \u0026amp; Algorithm leads working together, with Design in a supporting role. The \u0026quot;voice of the user\u0026quot; comes from understanding linguistic patterns, evaluation metrics, and iterative feedback loops.\nI’ve seen these patterns play out repeatedly on search, generative AI, and machine learning teams. If you’re trying to structure one of these teams differently, I’d be interested to hear what’s working for you. If you're looking for organizational help or a data and algorithms expert, I'm a freelance consultant with many years of relevant experience who can help your Search, Generative AI, and other teams be maximally effective. Get in touch!\nNote: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1770076800,"id":"15","link":"/2026/02/search-and-ai-teams-are-different/","section":"post","summary":"Engineering leadership knows the standard playbook for product teams: the Product Trio, the Spotify model, outcome-based roadmaps, and so on. I've seen teams adopt these practices and still struggle …","tags":["management","search","ai","software-engineering"],"title":"How Search and AI Product Teams are Different"},{"body":"If AI helps me get ideas from my head to the readers' heads faster, that's good. Society thrives when ideas are shared, critiqued, and built on. But authors shouldn't take credit for other peoples' ideas -- they should synthesize others' ideas, including the \u0026quot;blurry JPG of the web\u0026quot; that is LLMs. The question is: what makes AI-assisted writing defensible? When is it helpful to society, and when is it plagiarism? When can you still be considered the author of AI-assisted writing?\nAcademic writing has generally clear guidelines on how co-authorship and acknowledgements work. In the fields I worked in, people who edit or provide feedback on a paper only get an acknowledgement, not co-authorship. Primary authors and co-authors work together on the writing as well as the research behind it, often with a senior supervisory author cited last.\nFor AI-assisted writing, is the LLM the primary writer, an editor, a senior co-author, a junior co-author, or just something that deserves an acknowledgement? Obviously it depends, so how can we be clear to readers the role of the humans and AI? Which of these roles is \u0026quot;defensible?\u0026quot;\nThe Stakes Paul Graham writes that the expectation of writing in prestigious jobs historically forced individuals to learn how to think. Classical liberal education is a mechanism for improving thinking using writing, deep reading, and other techniques. If we outsource too much of the writing process, we risk losing that mechanism. Graham emphasizes the stakes of a shift to AI being a primary author by citing Leslie Lamport: \u0026quot;If you're thinking without writing, you only think you're thinking.\u0026quot; I'd add -- if you're writing without either thinking or writing, you're not really contributing anything.\nBut that doesn't mean we should avoid the productivity benefits of AI entirely. The question is how to use it defensibly.\nDefensible Patterns Seth Godin utilizes AI not to do his writing but to challenge it. He explains: you \u0026quot;say to Claude, please find the internal inconsistencies. Please ask me five hard questions. Please criticize the structure.\u0026quot; This use case is defensible because the human maintains agency. Godin suggests that the real opportunity is to \u0026quot;find a way to use human effort to create more value.\u0026quot; I like the motivation of creating more value, of getting my ideas into others' heads faster and better, and I now use AI to help me do all of my work faster, including writing.\nRyan Law asserts that \u0026quot;Generative AI struggles to provide any information gain.\u0026quot; A defensible use of AI, therefore, involves \u0026quot;front-loading the article structure.\u0026quot; By ensuring that the core of the content comes from a human with lived experiences, an AI can provide context and feedback, can strengthen arguments and improve how well the ideas are communicated. Conversely, relying on the AI to generate the core structure, then rewriting or adding to it, is less likely to yield novel or valuable writing. As I'll describe below, my process starts with a brain dump of ideas, then uses AI's ability to organize, more than synthesize, to start to convert those ideas into a coherent story.\nLaura Mohiuddin expands this into a \u0026quot;Defensible Content Playbook,\u0026quot; which shifts the focus from transactional blogging to \u0026quot;creating defensible, quotable intellectual property.\u0026quot; A threshold for value of professional content is that whatever you write should be good enough and novel enough for an AI system to cite you in the future.\nOn the far edge of defensible, Venkatesh Rao has an interesting blog that's primarily vibe-written. An LLM is the primary author, and he acts as a senior co-author, throwing out ideas for the AI to run with, providing feedback, but not actually typing the final version. I think that's defensible specifically because he's so transparent about the process. He generally includes a description of his prompts and processes at the end of each post. That's not what I do or want to do, but I appreciate it, and the posts are often thought-provoking.\nThe Joy of Creation Anil Dash says that the joy of creation is a fundamental reason for writing: \u0026quot;I write because it brings me joy. I'm not going to ask Claude to write my blog posts in the same way that I won't ask it to solve my daily New York Times puzzles.\u0026quot; For me, I enjoy getting my thoughts clear enough that I can write them down. I don't want to lose that part of the process, but the parts of writing that can sometimes be a slog, I'm happy to delegate.\nConversely, I enjoy software product development more than programming per se, so I'm happy to have the AI write code for me, both for my professional work and for hobby projects.\nMy Process Here's how I wrote this article, using AI for certain parts, to model one approach to transparency.\nMy blogs start on my personal web site, which is built in Hugo, a website and blog generator. (I will later copy the post to Medium for more readers.) I'm writing in Cursor, the AI-centric development environment and text editor. Cursor's Agent sidebar let's me have AI read or edit what I'm working on.\nMy first step is to use an AI command: /new-blog-post Defensible Use of AI in Writing (Like This). That generates the Hugo front-matter for my post, saving me a few minutes of boilerplate typing. I brain-dump bullet points into the document, in whatever order things come to me. Nothing is polished, but I want to get everything out of my head. I go to an AI with Deep Research capabilities (e.g., Google Gemini) and ask it to find me what else has been written on the topic, focusing on specific quotes and references over generalizations. Deep Research seems to me to be a lit-review put together by someone who doesn't really know what I want, and who writes in a very opinionated, very different style from me. I copy-paste some quotes and the URLs into my draft. At this point, I'd call this a first, very rough draft. I use another command, \u0026quot;rewrite blog outline\u0026quot;, to generate a readable second draft. The AI reads my prior (not AI-assisted) writing, and a summary of my writing style, then uses that to convert my notes into a reasonably well-structured blog post. I ask it to minimize wording changes, to maintain references, and to suggest opportunities for illustrations. I do a deep rewrite editing pass to make sure that everything reflects my views and approach. I use a \u0026quot;critique blog post\u0026quot; command to have the AI write three critiques of my article, one from a supportive reader, one from a neutral reader, and one from a critical reader. This is generally great feedback about my writing, and gives more opportunities to rewrite, tighten, and clarify. I've also been testing a humanizer skill that finds remaining signs of AI-written writing and tries to clean it up. I review and either accept or reject the suggested changes -- so far, it has been mostly helpful. I do one final re-read myself, then publish it. The AI isn't just editing -- it's helping with organization and suggesting connections and some of the content. But I'm maintaining control over the ideas, the voice, and the final product.\nWhat Makes It Defensible What makes AI-assisted writing defensible? Summarizing the above, I think we can state a few principles:\nTransparency: Be clear about how AI is being used, particularly if you're using it in a way that would necessitate co-authorship in an academic context. Human agency: Maintain control over ideas, voice, and final decisions. Information gain: Provide original insights, data, or synthesis that the AI couldn't generate on its own. Value creation: Use AI to help you get more of your ideas into more peoples' heads. Preserve the thinking: Don't outsource the parts of writing that help you think -- the initial brain dump, the synthesis, the final decisions. As noted above, thinking about the roles of contributors to academic writing may also be helpful:\nAcademic writing role Contribution AI credit? Notes First author main ideas, arguments, final wording definitely give full credit, ala Rao, or plagiarism Senior/final author suggested directions, major review, inspiration definitely should specifically call out contributions Secondary author limited writing, contributed to the work yes should note briefly Editor grammar, spelling, layout, workflow not needed Acknowledgement feedback on writing yes could be footnote/endnote If you're using AI to help you think faster or communicate better, while maintaining your voice and your ideas, that seems defensible. If you're using it to avoid thinking, or to plagiarize others' ideas (even if synthesized by an LLM) as your own, that's not.\nNote: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1769644800,"id":"16","link":"/2026/01/defensible-use-of-ai-in-writing-like-this/","section":"post","summary":"If AI helps me get ideas from my head to the readers' heads faster, that's good. Society thrives when ideas are shared, critiqued, and built on. But authors shouldn't take credit for other peoples' …","tags":["ai","writing","llms"],"title":"Defensible Use of AI in Writing (Like This)"},{"body":"What are Post-Query Refinement Suggestions? One of my favorite search UX patterns is post-query refinement suggestions — buttons that appear between the search box and results, adjusting the query in various ways. See, for instance, these suggestions on Etsy, which recommend that I filter by shipping speed, seller type, cost range, style, etc.\nFilter Suggestions on Etsy PQRS is a mouthful of an abbreviation, but most of what I've seen as alternatives either describes the UI element (\u0026quot;refinement pills\u0026quot;) rather than the function, or is excessively vague (\u0026quot;guided search\u0026quot;). So PQRSs it is, sorry.\nPQRSs differ from other search components. Auto-suggest and auto-complete help users finish queries before submitting. Faceted search is a full filtering interface using a taxonomy. On Etsy, faceted search appears when you press the Show Filters button.\nPQRSs work like a recommendation system, but applied to searches rather than items or categories. The semantics can vary: \u0026quot;did you mean?\u0026quot; suggestions to fix typos or reformulate without changing scope, or \u0026quot;narrow your search\u0026quot; suggestions. Here, I'm focusing on a specific case: \u0026quot;narrow your search by adding filters.\u0026quot;\nThe Problem: Overwhelming Search Results Suggesting filters helps in a couple of ways. It reduces friction when users try to narrow an overwhelming set of results to something manageable. It also helps communicate the range or diversity of items in the search results. Research suggests that users often struggle when presented with very long lists of results, or complex and new-to-them taxonomies. (I.e., the paradox of choice.) Compared with approaches such as faceted search, PQRSs are simpler to use, more curated, and less intimidating, but less flexible for power users. In many domains (e.g., e-commerce) that's the right trade-off to make.\nQueries that particularly need refinement are what Daniel Tunkelang has called broad and ambiguous queries. Broad queries (\u0026quot;shoes\u0026quot;) yield results that are too long to be practically reviewed, while ambiguous ones (\u0026quot;mixer\u0026quot;) might return several very different groups of results. Filters can address both of these, if only users could quickly identify which filter to apply.\nTechnical Benefits: Addressing Word-Proximity Issues PQRSs also address word-proximity issues. When users type multi-word queries, search systems prefer items where the phrase appears as-is or with few words between. For example, \u0026quot;blue sneakers\u0026quot; will preferentially find items where that phrase is used in the item, especially in the most-important fields like the item title. But it's highly dependent on how your item titles and descriptions are generated or worded. In one recent case, I helped a client address an issue where several brands had the equivalent of \u0026quot;blue sneakers\u0026quot; in the title, but one brand had \u0026quot;blue X123 sneakers\u0026quot;, with a product code. As a result, that brand was buried in the search results. If the user had used filters, with a query like color:blue category:sneakers, this issue would not have arisen.\nIn Algolia, this proximity issue can be partially mitigated by the minProximity setting, but allowing intermediate words (setting minProximity to 2 or more) can cause less-relevant matches to intrude in cases where the query actually was a phrase. There are trade-offs.\nBy highlighting useful filters, PQRSs reduce query length and proximity issues. Traditional query-suggest patterns add more search terms (typing \u0026quot;blue\u0026quot; suggests \u0026quot;blue sneakers\u0026quot;), not filters.\nGenerating Effective Suggestions How do you identify which filters to suggest to users? A typical e-commerce site may have a dozen or more facets (color, brand, size, style, price range, etc), each with many values that could be applied as filters. Search engines will happily tell you how many items in the result set have each facet value, but wading through a facet menu is time-consuming. And you don't want to give users a huge list of filters that they could apply -- Baymard has good research on this -- we want a tighter, more-curated alternative to faceted search.\nPQRSs are that alternative, if you can find a data source and algorithm that gives your users suggestions they actually want to use.\nI think there are four main sources for PQRS suggestions: 1) the result set, 2) interaction popularity, 3) business rules, and 4) personalization. The result set is is all items matching the query (retrieved, not necessarily ranked). Interaction popularity shows filters commonly applied by users — often effective. Business rules highlight categories the business wants to promote (high-margin items, new categories). Personalization uses the user's history to find relevant refinements. I'll be focusing on result-set suggestions here, but the others are also worth investigating and incorporating in practice.\nAlso worth noting: research has found that removing superfluous search terms can be helpful in search, too. One option is auto-filter. The UI can be almost the same, but instead of adding a filter, the system both adds a filter and removes the equivalent search term. This is the \u0026quot;did you mean?\u0026quot; pattern applied to query refinement. E.g., rugged shoes becomes rugged category:shoes. Recent work on AI-driven UX patterns discusses interesting smart auto-filter approaches.\nThe Demo Tool For a client, I recently built (well, vibe-coded, so designed and product-managed) a tool to help them see what various PQRS algorithms might look like on their specific catalog. That tool is of course work-for-hire, so I can't share it, but I've created a similar, generic demo tool, with no proprietary elements, that many Algolia customers can use to explore a few PQRS approaches for their catalog.\nAlgolia Post-Query Filter Suggestion Demo\nAlgolia Post-Query Filter Suggestion Demo A few features:\nInteractive setup with your App ID, API Key, and Index Name -- nothing is shared with me Visual algorithm comparison: switch strategies, see real-time suggestions and effects Easy customization: choose which fields to display, hide irrelevant facets Demonstrates click-to-apply filters pattern for faster result refinement Transparent algorithm logic -- step-by-step breakdowns for Coverage Diversity available in-app The app's About page gives more details.\nWhen properly configured, the suggested filters reduce ambiguity while clarifying the catalog structure. All of the algorithms in the demo are based on the result set -- the top items that were returned as well as the facet counts. Following the principle that users should understand what's happening, the demo does not auto-apply any filters -- users can see exactly what's happening to their search. (Algolia can be used to set up auto-filter rules, but they are generally opaque to the user, which can cause problems if the filters are applied improperly. Query Categorization is another related Algolia feature, which can be used to implement a type of PQRSs, or to auto-filter.)\nAlgorithms I implemented four algorithms for this demo app: popularity, information gain, inverse frequency, and coverage diversity. There are many more algorithms and variations possible -- these were just easy to implement and represent different approaches.\nPopularity simply suggests the most common facet values in the result set as filters. This sort of popularity is distinct from which filters users apply most often -- both have value. But suggestions can be redundant with the query or with each other, so popularity alone isn't ideal. Information gain suggests filters that cover as close to 1/2 of the result set as possible. It's good for efficiently narrowing the results, but often can be unexpected. Inverse frequency suggests uncommon facet values, which is rarely useful on its own, but is worth seeing for comparison.\nCoverage diversity is my preferred primary approach -- it suggests a diverse set of facet filters that together cover the result set well. Intuitively, it finds a filter that \u0026quot;carves off\u0026quot; about half of the items in the result. Then, it looks at the remaining items, and tries to find another filter that applies to half of the remainder. Compared with Popularity or Information Gain, Coverage Diversity identifies non-redundant filters. For an Algolia implementation, only the top items are available, so the algorithm uses those items as a proxy for the full set.\nDemo app\u0026#39;s explanation of coverage diversity algorithm. In the example above, from an Algolia demo dataset, where the user typed \u0026quot;blue\u0026quot;, the algorithm identifies that more than half of the top results are from a Brand called BLU. Of the rest, many are Unlocked Cell Phones. Clicking either of those buttons would apply the filter and re-run the algorithm with the resulting, narrower results.\nCoverage diversity is related to Maximum Marginal Relevance (MMR), which diversifies search results by penalizing items similar to already-shown ones. (Elastic has a good explanation). MMR applies to items instead of facet values, but has similar intuitions.\nIn practice, combining coverage diversity with other approaches will likely work best. The exact algorithm depends on the contents and size of your catalog, your taxonomy, and the expertise, biases, and patience of your users. This tool is a good first step -- if what you see is promising, vibe-coding something specific to test your chosen algorithm(s) is quick.\nAdvertisement: Does your organization struggle to get the relevance, user experience, and business impact you need from Algolia? I'm a freelance consultant with years of Algolia experience who can help you get the most out of advanced tools and search algorithms. Get in touch!\nNote: This post was primarily human-authored, with AI assistance for research, editing, and organization. The AI filled a Secondary author role. The core ideas and final voice are mine.\n","date":1769126400,"id":"17","link":"/2026/01/post-query-refinement-suggestions-in-search-ux/","section":"post","summary":"What are Post-Query Refinement Suggestions? One of my favorite search UX patterns is post-query refinement suggestions — buttons that appear between the search box and results, adjusting the query in …","tags":["search","ux","product-design","algolia","algorithms"],"title":"Post Query Refinement Suggestions in Search UX, and an Algolia Demo App"},{"body":"I keep seeing AI skeptic takes like \u0026quot;AI is useless\u0026quot; or \u0026quot;LLMs are only good at Natural Language Processing\u0026quot;, usually because of hallucinations and AI slop. Both are real problems. But that conclusion still misses the point: it misidentifies what LLMs are actually good at.\nMy take is: LLMs are great at translation (in the broad sense), and they have emergent but fallible reasoning. Knowledge, in the \u0026quot;just tell me what's true\u0026quot; sense, is not their strong suit, despite how convincingly they can sound.\nTranslation: what LLMs are actually for Translation was one of the original NLP tasks that motivated their creation (literally: translating between natural languages). But \u0026quot;translation\u0026quot; is a lot broader than Spanish-to-English. It's taking information in one form and turning it into another form.\nOnce you start looking for it, a huge number of practical problems are just translation tasks. Programming is translating from \u0026quot;what I want\u0026quot; into working code, which is why generative AI feels most obviously useful there.\nQuestion-answering with Retrieval-Augmented Generation (RAG) is also translation. The LLM is turning retrieved snippets into the answer shape the AI Engineer asked for. The LLM gets context from somewhere else, and it turns that into a response. That's not \u0026quot;the model knows\u0026quot;; it's \u0026quot;the model can rewrite and synthesize what's in front of it.\u0026quot;\nAs an aside: consumer tools like ChatGPT and Claude often do pull in web or other data sources behind the scenes. So yes, you can ask factual questions and often get decent answers. But that's because the product is a bundle of systems (retrieval + tools + orchestration), not a naked LLM sitting in a box.\nReasoning: the surprising emergent capability The big surprise was that this translation machine could also do reasoning. Emergent abilities appear as models scale up, showing capabilities that weren't explicitly trained for. The most plausible story I've heard is that the training data contains lots of examples of human reasoning: explanations, proofs, arguments, step-by-step solutions. The model learns those patterns and can use language to \u0026quot;think through\u0026quot; problems, imitating step-by-step progress. Techniques like Chain-of-Thought prompting leverage this by explicitly encouraging the model to show its work.\nBut it's not formal reasoning, like decades-old computational systems that use provable algorithms to guarantee correctness. LLMs don't have that guarantee.\nThat said, human reasoning is also a messy mix of pattern-matching and approximation. Heuristics and biases research, from Tversky and Kahneman's foundational work onward, shows that humans use mental shortcuts and pattern matching when reasoning, leading to systematic errors. Human reasoning isn't formal and provable either. It's approximate and fallible.\nLLMs often make mistakes that rhyme with human reasoning failures: confirmation bias, jumping to conclusions, missing edge cases. They can conflate content plausibility with logical validity, accepting plausible-sounding arguments that aren't logically sound. Still, they're remarkably good, just as smart humans are at reasoning in domains they know well, despite making errors.\nIt's worth noting that both humans and LLMs use external tools to increase their reasoning capabilities. Humans use pen-and-paper, calculators, and formal systems for reasoning they can't do reliably in their heads. LLMs can use similar tools -- calculators, code execution environments, formal logic systems, or retrieval augmentation. The parallel suggests the gap between human and LLM reasoning might not be as fundamental as it first appears.\nTaxonomies of Understanding Another lens on what LLMs are good at is Bloom's Taxonomy, which categorizes levels of learning and understanding from basic to advanced:\nBloom\u0026#39;s Revised Taxonomy (source)\nRemembering: This is the most basic level, and LLMs are oddly shaky here. The classic hallucination is a confident-sounding \u0026quot;fact\u0026quot; that's just wrong. Understanding: Organizing and summarizing information. They're excellent at this. Applying: They can often solve problems in new-but-analogous domains. In software engineering, translating between specs and code is where they shine. In less-structured work, \u0026quot;apply\u0026quot; can be hit-or-miss but often still useful. Analyzing: They're strong at proposing possible causes, motivations, and structure, especially as a starting point. Evaluating: Mixed results, especially when evaluating their own output. In production systems, \u0026quot;evals\u0026quot; (and sometimes one model critiquing another) help, but it's not magic. Creating: Originally called \u0026quot;Synthesis\u0026quot; by Bloom. This is another place where LLMs often disappoint: superficial synthesis is easy, but genuinely new, coherent synthesis is harder than it looks. It's interesting that LLMs excel at the intermediate levels of this taxonomy but fail more often at the bottom and top levels. That's not a human-like pattern. LLMs are not human intelligences.\nWhat this means If you keep \u0026quot;translation + fallible reasoning\u0026quot; in your head, the practical guidance gets simpler.\nUse them for translation-shaped tasks: converting between formats, generating code from descriptions, turning context into answers. Use them for reasoning when you can tolerate mistakes, when a human will review, or when the cost of being wrong is low. And whenever you can, let them use tools (retrieval, calculators, code execution) instead of pretending they have perfect internal knowledge.\nWhat I try not to do: treat them as context-free knowledge bases, no matter how \u0026quot;sure\u0026quot; they sound. I also don't rely on them for formal correctness, or for situations where one bad answer is a real problem. And for the most creative work that requires deep, coherent synthesis, people are still better.\nCritically, understand when you're working with a raw LLM vs. a multi-system AI with fewer limitations. Know what you're working with, and match the tool to the problem.\n","date":1768953600,"id":"18","link":"/2026/01/use-llms-for-translation-and-fallible-reasoning/","section":"post","summary":"I keep seeing AI skeptic takes like \u0026quot;AI is useless\u0026quot; or \u0026quot;LLMs are only good at Natural Language Processing\u0026quot;, usually because of hallucinations and AI slop. Both are real problems. …","tags":["llms","ai"],"title":"Use LLMs for Translation and Fallible Reasoning"},{"body":"Algolia is a tremendously powerful search platform, especially for e-commerce. But like most SaaS tools, the management dashboard doesn't do everything you need it to do. In particular, it's hard to compare search result rankings across variations in indexes -- a super common pattern when you're testing and iterating on index configurations before A/B testing, implementing algorithm and UI proof-of-concepts, or refining search relevance.\nAt several places I've worked, I've built (or spec'ed out) comparison apps that allow you to run the same query in parallel and easily see and compare the results, for internal testing. Most recently, I vibe-coded an app that has worked very well. I can't share the code itself, so I've written a spec for a generic, equivalent app that is free for anybody to use and customize.\nThe app includes several key features that make it useful for this sort of testing:\nMultiple-column display, highlighting position changes of results Ability to configure and select multiple indexes and apps Ability to configure display mappings Filter support Info popup with ranking debugging info Mobile-friendly design Algolia Comparison Dashboard The spec is in OpenSpec format, and you can use your favorite AI coding tool (I'm partial to Cursor CLI, but I'm sure Claude Code would work well too) to implement it. Any changes you make to the spec before you build, such as changing web frameworks, will be automatically incorporated into the generated code. The spec and instructions are available on GitHub -- you can use it to get started building your own comparison tool.\nOut of the box, this is helpful for testing configuration changes, such as Algolia replicas with different index parameters or tiebreakers. But I've also used it as a test-bed for a variety of algorithms, which I spec'ed out and had the AI write. For example:\nRerank the top items to increase the number of brands visible on page 1. (Avoid the \u0026quot;white shoes\u0026quot; being all-Nikes problem.) Auto-apply filters in a non-magical way. (\u0026quot;white shoes\u0026quot; becomes \u0026quot;color:white category:shoes\u0026quot;, but with filters visible and removable by the user) Post-query refinement suggestions. (Filters that helpfully narrow down very long results.) Adding features like this, at least enough to test and validate, can take the AI just five or ten minutes, if you have a clear idea of what you want to build. It's now faster to try it out and throw it away than to schedule a meeting to discuss your idea!\nAlgolia users need great, custom tools for testing and iterating on their index configurations. Vibe-coding lets you build these tools quickly -- and now that it's so much easier, there's less of an excuse not to have them. Implementing a high-quality search experience relies on having great, custom internal tooling.\nAdvertisement: Does your organization struggle to get the relevance, user experience, and business impact you need from Algolia? I'm a freelance consultant with years of Algolia experience who can help you get the most out of advanced tools and search algorithms. Get in touch!\n","date":1768262400,"id":"19","link":"/2026/01/vibe-coding-missing-algolia-comparison-dashboard/","section":"post","summary":"Algolia is a tremendously powerful search platform, especially for e-commerce. But like most SaaS tools, the management dashboard doesn't do everything you need it to do. In particular, it's hard to …","tags":["algolia","vibecoding"],"title":"Vibe-Coding the Missing Algolia Comparison Dashboard"},{"body":"The rise of AI-assisted coding tools has made it dramatically easier and faster to build small side projects than ever before. What used to take weeks of evenings and weekends can now be done in a few focused sessions, with an AI pair programmer handling much of the boilerplate, debugging, and implementation details. This has opened up new possibilities for building tools that solve personal problems or scratch specific itches, without the overhead of building a full commercial product.\nI've been taking advantage of this myself, building a few small projects over the past year. One is a Scrabble game for me and a few friends -- I can't publish it publicly for copyright reasons, but it's been fun to have a custom version that works exactly how we want it to. Another is a Chrome extension that makes the New York Times website more customizable and less annoying (at least to me). Expect me to share more about that soon. And now I'm ready to share the third: OpTree.\nOpTree is a tool for building Opportunity Solution Trees -- a methodology for structuring product discovery. It transforms the product discovery process by creating visual trees that connect outcomes to opportunities, solutions, and experiments, all in a collaborative workspace. Inspired by text-to-diagram tools like Mermaid, OpTree brings that same philosophy to product discovery: you define your tree structure in text, and it renders in real-time as a visual diagram. Features include comments, solutions for multiple opportunities, priorities, and more. You can check out the website for more details on what it does and how it works, then use it yourself!\nOsTree Editor Syntax Rendered OsTree I'm releasing OpTree as donationware -- free to use, with an optional way to contribute if you find it useful. This isn't a commercial product, and I don't plan to grow it into one. I built it primarily for my own use, and I'm sharing it in case others find it helpful. If a few people chip in to cover server costs, great. If people suggest improvements, even better. But I'm not trying to build a business around it.\nWhy donationware instead of open source or a commercial product? For this particular tool, open source doesn't quite fit -- I want to maintain control over the direction and keep it simple, without the overhead of managing a community or dealing with forks and contributions. And I don't want the pressure and complexity that comes with trying to build a sustainable commercial product. Donationware feels like the right middle ground: it's available to anyone who wants to use it, there's a low-friction way to contribute if people want to, but there's no expectation of ongoing support or feature development beyond what I personally need.\nI hope we'll see more small, public donationware tools like this emerge, now that it's so much easier to build and deploy small software systems. Not every tool needs to be a startup or an open source project. Sometimes a simple solution to a specific problem, shared freely with an optional tip jar, is exactly what's needed.\nFor what it's worth, I've been doing most of this AI-assisted coding work in Replit, mostly because deployments are so easy and built-in. But I've also used Cursor combined with GitHub Actions and hosting platforms like Fly.io for other projects. The tooling matters less than the fundamental shift: AI assistance has made it practical to build and ship small tools that solve real problems, without the traditional barriers of time, complexity, and deployment friction.\n","date":1767916800,"id":"20","link":"/2026/01/ai-assisted-coding-side-projects-and-donationware/","section":"post","summary":"The rise of AI-assisted coding tools has made it dramatically easier and faster to build small side projects than ever before. What used to take weeks of evenings and weekends can now be done in a few …","tags":["ai","coding","side projects","donationware"],"title":"AI-Assisted Coding, Side Projects, and Donationware"},{"body":"Consulting I help growing technology businesses turn AI, machine learning, search, and data into revenue, product quality, and better governance.\nWhat I Do I'm a data science leader with a PhD in Computer Science (AI/ML) and 15+ years of industry experience across EdTech, e-commerce, marketplaces, and enterprise software. Here's where I go deep:\nAI \u0026amp; LLM systems — I move AI from prototype to production. At EdSights, I replaced an obsolete chatbot with an LLM-based microservice, delivered on time and on budget, doubling student engagement. Search \u0026amp; Recommendations — I build high-impact search and discovery systems that drive revenue. At Teachers Pay Teachers, I improved search relevance and marketplace discovery, increasing search-driven revenue by ~5%. Data \u0026amp; Analytics infrastructure — I turn underperforming data functions into trusted analytics engines. At Algolia, I built an Analytics Engineering team responsible for trusted data, BI standards, and stakeholder alignment across the business. Team development \u0026amp; hiring — I help companies close talent gaps and build Data Science, AI, and Search organizations that scale. How I Work I'm a freelance consultant based in New York, NY (Eastern Time). I work hourly, with flat-rate options for well-scoped projects. I'm equally comfortable in discovery and strategy, or hands-on writing production-grade code. I'm available for advisory engagements, as well as interim or fractional Head of Data, AI, ML, or Search roles. Ready to Talk? Start a conversation today:\nOn LinkedIn Or just drop me a line ","date":1762522860,"id":"21","link":"/consulting/","section":"","summary":"Consulting I help growing technology businesses turn AI, machine learning, search, and data into revenue, product quality, and better governance.\nWhat I Do I'm a data science leader with a PhD in …","tags":[],"title":""},{"body":"As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we can learn from statistics. I (and many, many others) have found that there is a lot of value in having a document, shared among the team that is running the test. Some A/B testing tools include some limited workflow for collaboration on testing (e.g., Amplitude Experiments has pre-test, monitoring, and results tabs with useful graphs and summary statistics), but I think the most important thing is to have a tool that helps the team get aligned by forcing them to write things down.\nI'm including below a template that I've used in the past. It's a simple collaborative document that you could put into Google Docs or Confluence or Notion, and just make copies for each A/B test. It is intentionally not part of the A/B testing tool, as it needs to be accessible to everyone, including external business stakeholders, in a familiar way. I'm going to annotate each section with some motivation -- why I think that it's important for the team (say the Data Scientist or Analyst, Product Manager, and Engineers on a team) to all work together on this document.\nDesign 🧪 Problem statement 🔎 What pain are we solving for our users? What are we trying to learn? Motivations and assumptions behind this change?\nIn terms of the structure of this document, first of all, emojis are important. They add an element of how you're supposed to feel in each section, which is motivating. Plus they're fun. Second, having these italicized prompts I've found to be super-helpful. Otherwise it's too easy to just type in a few words without thinking through why we're doing what we're doing.\nFor this specific introductory section, it's important for everyone to understand not just what we're doing, but why -- why we think a specific change may help users, or tell us something about their desires and behavior. Also, asking for assumptions is important, as the process of identifying them may raise issues about the design of the experiment even before it starts, which will save time!\nData 📊 Add supporting insights here. What do we already know about user behavior?\nMost A/B tests don't happen in a vacuum. Maybe we've run previous tests, or interviewed users, or analyzed funnels. What numbers do we have already that will ground the results? Do we have data that makes us optimistic about a change?\nHypothesis 💡 We expect that by changing X, we will cause user behavior to change to Y, yielding metrics impact Z.\nThis is critical -- we need to have a guess about a mechanism that we're changing, and we need everyone to understand it. It also helps the team understand what's plausible, in terms of outcome metrics.\nPrimary Metric 📈 What metric will we use to make a decision, once the experiment is complete? (link to a graph of the metric's recent historical trends) (link to prior experiments that attempted to move this metric)\nEveryone involved in an A/B test needs to have good intuitions about the primary metric that you're trying to move. How much does it vary over time? What causes it to vary? How much has the team been able to move the metric in the past? Dropping in these links, and reviewing them frequently, help develop this intuition.\nSecondary and Guardrail Metrics 📈 What other metrics will we be tracking? What metrics might block a release, if they decrease substantially, even if the primary metric is as we hoped?\nI highly recommend Kohavi et al.'s treatment of Guardrail metrics, based on decades of collective experience running A/B tests. It is likely that changes will introduce unintended side-effects, so you need to track many things in almost any A/B test.\nVariants 🆎 Add visuals and description of the control experience and all variants, and/or link to the PRD.\nSome A/B tests will be supporting a change that has been described previously in a Product Requirements Document. Others will be simple cosmetic changes, where just screen shots of what's changing are adequate.\nTargeting 🎯 Which users will be part of this experiment? Who should be excluded? (platforms, segments, etc) Will the analysis be segmented? (That is, will any user sub-groups be looked at separately?)\nYou need to agree on this in advance. Although you should do post-hoc analyses, to learn as much as possible, you really want to make decisions based on criteria you've determined in advance. Who is part of the experiment, and what differential impact on different sub-groups might mean, needs to be part of that discussion. For instance, what if your change increases a metric overall, and for new users, but hurts it for returning users? Do returning users matter more? This ties into the next section:\nDecision Plan ⚖️ What outcomes will lead to what decisions? Who is involved in the decision, and who is the final decision-maker? Is this a Superiority, Bias-to-Ship, Agnostic, or other type of decision?\nSee The Five Types of A/B Test Decision for more on these types.\nIt's important to know who will be making the call, and to plan ahead for various scenarios. What if your results are ambiguous? What if a key guardrail metrics is trending down? The more you can plan ahead, the faster you can move once data is in. And fast decisions is the whole game.\nRollout 🚀 Allocation 💯 How will we split users and/or ramp up?\nOff group (not in the experiment)\nVariant name: off Percent: Off_Control group (in the experiment, control condition)\nVariant name: off_control Percent: On group (in the experiment, test condition; add more groups if testing more than one thing)\nVariant name: on Percent: This is an important technical detail. If you're planning to ramp up the experiment, perhaps first exposing a small proportion of users to ensure that the code is working properly, then you must have a three-condition experiment. Users who were in the Off group initially should get re-bucketed to Off_Control or On randomly, once you've ramped up, but the users who were in those groups initially should not get re-bucketed.\nIf you don't, and you start with (say) a 5/95 split, then move to 50/50, you lose several things. You can no longer easily determine if your A/B split is working correctly. To the extend that the effect of your change varies over time (perhaps weekends vs weekdays), you may well end up with biased and hard to understand results. And some users may switch from Off to On, which can also confuse your results. I believe that all standard A/B testing tools handle this correctly, but be careful if you're using a home-grown feature-flagging system.\nPower Analysis ± (may be customized by data science, depending on the experiment design and decision plan)\nEstimated weekly traffic exposed to the experiment (all variants):\nDecision requirements for this metric:\nNumber of weeks required:\nLink to analysis:\nBefore the experiment, a data scientist (or well-trained analyst) should analyze the key metric and how it's expected to be affected, and perform a power analysis. This could be as simple as using an on-line calculator, or it may require a custom analysis and simulation. The \u0026quot;decision requirements\u0026quot; are based on the type of A/B decision (see the Five Types post), but are generally a threshold for a change in the metric. It's important that everyone agrees on how long the experiment is to be run for.\nMonitors 🚨 How will we ensure that data is flowing, the A/B split is correct, etc? link/embed tracking charts\nThis could be to the A/B testing tool you're using, or to a dashboard based on your data warehouse. But regardless, it's important to peek in those first few days, to make sure that everything is working as expected. But don't expect to learn much if anything about your metrics -- those will bounce around a lot before settling down!\nI generally recommend only peeking to determine if you need to stop an experiment that's actively hurting metrics. A good time to do that is often after 1 week, when you've had a full weekday/weekend cycle. Otherwise, waiting until the planned duration has completed will maximize the likelihood that you make the correct decision.\nLearn 🎓 Experiment Results 📈 Impact to primary metrics for test. Describe and show results, then fill out the list.\nThis is where you'll describe what happened. I've found it useful to put charts and graphs and statistics here, but it's also important to have a standard, consistent format for the results:\nDates: January 1, 1970 - January 15, 1970\nOEC: Furlongs per Fortnite\nOutcome Consistent Interval: -10.0% - +5.0%\nDecision: Roll back\nWhen did we run this test, what was the Overall Evaluation Criteria, what was the uncertainty/confidence/plausible interval on the change to the OEC, and what did you decide to do. Every test must have something like this. It's often a good idea to export this table into a spreadsheet or something, so you can easily see the results of all of your tests. (In Confluence, you can do this sort of automatically with Table Excerpt macros!)\nInsights 🕵️ Secondary metrics and emergent behavior. Other learnings such as usability, segment trends, traffic shifts, etc.\nYou didn't just make a decision, you probably learned something about your users. That's valuable information for your company. Write it down.\nNext Steps ⏭️ Action items, such as roll out winner, additional user testing, iterations on winning (or losing) variant, retest, etc.\nAlmost every test I've been a part of has led to, at a minimum, ideas for other things that could be changed.\nAppendix 📖 Resources and notes 📝 Any links to Figma, Jira, Docs etc?\nDevelopment details 💻 Worrisome edge cases, dependencies, tradeoffs made, teams impacted -- link to technical design doc(s)\nSome A/B tests are easy to implement. Others aren't, or have downstream impact. You probably want to document this.\nThat's the document! You are more than welcome to use it within your organization. Please let me know about your experience, whether having a consistent, non-jargon-y template helped your team get aligned, and whether you ended up making useful changes!\n","date":1697068800,"id":"22","link":"/2023/10/a-collaborative-template-for-a-b-tests/","section":"post","summary":"As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we …","tags":["a/b testing","product","data science"],"title":"A Collaborative Template for A/B Tests"},{"body":"The other day, I was reading a post by Venkatash Rao (thousands of words of under-edited brilliance, as usual), and was struck by this note about the complexity of climate solutions:\nI tend to take as an article of faith the systems science rule of thumb that the complexity of solutions generally matches the complexity of the problems. If it doesn’t, then you either got lucky, or there are negative externalities you’re ignoring.\nThis resonates with some discussions I’ve had recently about the application of Large Language Models (LLMs) and related AI technology.\nSimplicity, Complexity, and Code Einstein (and many other people) have said that things should be as simple as possible, but no simpler. This applies to technology businesses in a number of ways. Larger companies can do more things, but have coordination costs as the business involves more people and gets more complex. In software design, simpler architectures are preferred because they’re less fragile and easier to reason about and debug, but abstractions allow easier major changes, better scaling, and more ability to meet diverse user needs, even at the cost of more lines of code. When trying to move quickly, software organizations usually incur technical debt -- they implement solutions by layering complex logic on existing code, resulting in excessive complexity. Or else they implement half-solutions, resulting in overly-simple applications that don’t match the complexity of the problem. Addressing this technical debt involves rewriting code to make it the right level of complexity. (See also Mike Loukides’ insightful recent notes on complexity.)\nIn the field of AI, an important paper was written in 2014 by researchers at Google: Machine Learning: The High Interest Credit Card of Technical Debt. They describe a number of problematic patterns when applying ML systems, which intrinsically have complex feedback loops and external dependencies, to software systems. This paper led many practitioners to be very focused on business value when considering adding ML systems to a product, and to avoid flashy/trendy solutions unless they clearly will move the needle substantially. Even so, ML models need to be proactively monitored for “drift” (decreasing quality over time), and it’s typical to have to re-train each model a couple of times per year in common business applications. Regardless of the number of lines of code written, ML systems are complex, with substantial maintenance costs.\nSome Questions About LLMs Jump ahead a few years to 2023. Essentially every organization larger than a corner laundromat is now trying to figure out how to apply the new AI technologies of LLMs and related systems. Are these tools complex, or simple? In what ways? Plugging into a vendor’s API seems simple, but as Venkatash Rao would ask -- are you lucky, or are there negative externalities that you’re ignoring that will come back to bite you later?\nI mostly have more questions, rather than answers, but I think they’re questions that need to be addressed when you’re evaluating how and when to implement LLMs as part of a product or system right now.\nLet’s start with baselines. When implementing complex predictive models, it’s usually a good idea to compare your solution with literally the simplest thing you can think of. Want to categorize text? Ask an expert to give you 3 words that make it pretty likely that the text is in the category of interest, and spend the ten minutes it might take to build that logic. Measure how effective that simple logic is. Then prototype the fancy solution -- does the machine learning model give you enough of a step change to be worth incurring the substantial complexity and maintenance costs? Sometimes it does -- great!\nAsk the same question about LLMs. What’s the baseline? Implementing an LLM either requires being locked into an expensive vendor, or deploying your own expensive and temperamental system. Can you do something much similar, and presumably much worse, that will give you baseline metrics?\nHow will you monitor your LLM system? Just like any other ML system, it will likely get worse over time, as the system itself changes (especially if you’re buying from a vendor), and as the world changes. How will you handle drops in performance? Do you have a fallback that you can put into place immediately if the LLM becomes problematic? LLMs are stochastic -- they give different results for the same input randomly -- how will you develop confidence that it’s working well-enough?\nMaybe you already have an existing “dumb” system that you’re considering replacing with an AI system. What risks are there? Sure, you could try to replace an existing user-facing system with a chatbot, but what happens when someone asks it “how to I make napalm out of your product” and posts screenshots to social media? What if people stop being enthralled by the novelty of typing questions into a box and getting mostly-good results, and start going to your competitors who require less typing?\nMaybe there are some less risky options than fully replacing large systems? Can you break the problem into pieces, and identify if any of the pieces could be replaced in a way that would be dramatically better with less downside? Do you have high-value employee-in-the-loop processes that would benefit from smarter suggestions? (Keeping in mind that humans will tune out when reviewing the work of automated systems that are almost always correct. This is why \u0026quot;human-supervised\u0026quot; levels of self-driving cars don’t work well.)\nOr maybe, cynically, the problem you’re solving is to put an \u0026quot;AI-Powered!!1!\u0026quot; checkbox on the marketing page of your website. That’s annoying. How can you introduce LLMs slowly, in limited situations, with lots of testing, so that you can keep your sales team from technically lying (more than they already do)? Can you keep it simple and minimal?\nIt’s worth noting that the currently most-valuable application of these tools seems to be helping programmers write code and learn how to write code better. As Gary Marcus says, \u0026quot;coding is a solid use case... programmers will never go back\u0026quot;. Programming is a complex problem, where the complexity is warranted, where attentive humans are in the loop, and where the risk is primarily that the tool doesn’t work, and humans will have to program the computer from scratch sometimes. For writing code, it seems like most of the questions above have straightforward answers. But that may be the exception, not the rule.\n","date":1694649600,"id":"23","link":"/2023/09/are-llms-a-simple-solution/","section":"post","summary":"The other day, I was reading a post by Venkatash Rao (thousands of words of under-edited brilliance, as usual), and was struck by this note about the complexity of climate solutions:\nI tend to take as …","tags":["llms","simplicity","machine learning","product"],"title":"Are LLMs a simple solution? And if so, for what problems?"},{"body":"There’s been an immense amount of discussion about Large Language Models (LLMs) such as ChatGPT over the last year, of course. Some of that discussion has been whether they are intelligent, conscious, or on the path to Artificial General Intelligence.\nI’m particularly interested in the \u0026quot;consciousness\u0026quot; question, as it was an area of personal interest when I was working as a cognitive scientist, in a prior career. I never did research on the topic, but I read plenty of philosophers of mind and neuroscientists as they tried to pin down what, exactly (or even vaguely), consciousness might be. One of my favorite treatments of the topic is by neuroscientist Antonio Damasio, most accessibly covered in his book The Feeling of What Happens. (I recently read a new collection of short essays by him, entitled Feeling \u0026amp; Knowing, but I wouldn’t recommend it. It feels more like footnotes or stray thoughts than a coherent presentation or a novel contribution.)\nDamasio makes a number of claims, a few of which I’ll try to briefly summarize here. I think that it’s worth thinking about LLMs in the context of theories of mind and consciousness, to clarify what those systems are actually doing (or not doing), and what it might take to tentatively bridge the gap. In some ways, LLMs seem conscious, in that they behave in a way that, if a human did that, we would say that they were conscious.\nDamasio’s core ideas are as follows (Wikipedia has a good summary):\nIntelligence is a low bar. Any system that reacts in a self-beneficial way to both external stimuli and what he calls \u0026quot;homestatic\u0026quot; signals is behaving intelligently. Homestatic signals are internal body signals about the chemical and physical health of the body, such as whether blood chemistry, temperature, or other systems are within bounds, or if the body has been damaged. Single-celled animals are, under this view, intelligent. Consciousness is based on a lower-level capability, which is the ability to represent (remember) the homeostatic signals in some form, rather than just react to them. When humans have a feeling that they are cold, or thirsty, or in pain, then they are representing these signals as having a relationship to the self. So, it seems likely that most animals are conscious. Human-level intelligence and consciousness requires other systems, such as language, attention, and theory-of-mind. But those things aren’t required for basic consciousness. David Chalmers, a philosopher and another prominent writer in the field, has recently been writing about LLMs and consciousness. He doesn’t believe it’s likely that current LLMs are conscious, and provides a half-dozen gaps that he suspects rule it out. But these gaps could, in theory and likely practice, be addressed.\nThree of his gaps, two that he spends a few paragraphs on, and one that he mentions in an \u0026quot;and also\u0026quot; list, that I think get at the heart of what might be required to build a minimally-conscious LLM model:\nFirst is a \u0026quot;global workspace\u0026quot;, a \u0026quot;limited-capacity... central clearing-house in the brain for gathering information from numerous non-conscious modules and making information accessible to them.\u0026quot; LLMs don't really have this, although systems such as ChatGPT have limited contextual memories that start to move in this direction. Second is \u0026quot;unified agency\u0026quot; -- a conscious entity needs \u0026quot;stable goals and beliefs of their own over and above the goal of predicting text.\u0026quot; ChatGPT has constraints about what it's willing to generate, but its responses don't generally appear to be from a single entity. Third is \u0026quot;stimulus-independent processing... thinking without inputs\u0026quot; -- you might call this a stream of thought or even a stream of consciousness. When nobody is asking a LLM a question, it's basically comatose. No processing is going on at all. It might be that LLMs, because of their general-purpose nature, and their ability to summarize prompted text relatively well, could be used in a relatively simple system with these three capabilities and rudimentary consciousness. People have been building more-complex LLM-based systems with aspects of this already -- Chain of Thought prompting and Tree of Thought architectures already get at the idea of having processing be based on the composition of multiple prompt-response pairs. The goals of these systems has generally been to improve response accuracy, especially for reasoning tasks.\nWhat if we focused on the consciousness-related gaps, rather than performance on any specific task? What would a simple LLM-based system with a global workspace, unified agency, and a stream of thought look like?\nHere's my completely untested proposal for such a system.\nThe system maintains a queue of something like 20 (50?) text statements, generated as described below. This is the global workspace. Old entries disappear as new ones are added. The system asks prompts of its own global workspace, while also reacting to external prompts (potentially from multiple external sources/people). These prompts happen on a regular basis, perhaps every 10 seconds, regardless of whether external prompts are happening or not. Those system-generated prompts might have several standard forms, perhaps something like: What are the key things a conscious, intelligent agent motivated by helpfulness, kindness, and a strong desire to avoid harm to humans or humankind (abbreviated CIAHKAHH, for the purposes of this blog post, but presumably spelled out in practice) would remember from the global workspace below? What would a CIAHKAHH feel about the global workspace below? What goals would a CIAHKAHH have, given the global workspace below? External prompts might get prefixed with the most recent responses to these questions: You are a CIAHKAHH with the following recent history: [global workspace]. You have been asked the following question by human X. Please respond accordingly. [prompt] The result would be summarized before being added to the list: You are a CIAHKAHH with the following recent history. [global workspace] Please summarize the below prompt and response into a two sentence summary to be added to your recent history. X prompts: [prompt] You responded: [response] The unified agency is maintained by the mix of hard-coded goals in the standard prompts, and dynamic goals maintained in the global workspace. Different instances of this system would have different dynamic goals, based on experience, and thus different but unified agency. Stimulus-independent processing, the periodic self-referential queries of the global workspace, is key to making this all (I suspect) work reasonably coherently.\nProposed LLM-based perhaps-conscious architecture Are there independent reasons for having a consciousness system like this? That is, would consciousness be adaptive for an intelligent system like a language model? It might! In particular, by having these hard coded goals (I’m reminded of Asimov’s 3 laws of robotics), a model like this might give more helpful answers, and might be less likely to provide problematic responses.\nWhat about risks? Unlike other approaches to extending LLMs, such as embedding/embodying them in the real world, or providing them access to external systems with real-world effects, it’s hard to imagine a system like this causing a great deal of harm. Certainly, with the hard-coded goals, it’s less likely to do so than a simple Google search, which will happily tell you how to make home-made napalm. (Please don’t.)\nAnd what about obligations? Would it be murder to pull the plug on a system like this? I don’t think so. Most philosophers of mind are pretty sure that animals such as mice have consciousness (although almost certainly not based on any sort of language). Most humans set out mouse traps and eat relatively intelligent and conscious mammals without much guilt, and certainly without criminal charges. Also note that just because this system is based on language, doesn’t mean that it rises to human-level consciousness. It just so happens that we invented LLMs that are good enough to implement this sort of thing before we invented the sort of highly complex multi-sensory systems that likely are required for consciousness in (say) mice.\nDoes this proposal make sense? I’m not deep in the cutting edge of LLM-based cognitive science right now, so if I’m restating ideas other people are already running with, or have already ruled out, please let me know!\n","date":1693699200,"id":"24","link":"/2023/09/llms-and-theories-of-consciousness/","section":"post","summary":"There’s been an immense amount of discussion about Large Language Models (LLMs) such as ChatGPT over the last year, of course. Some of that discussion has been whether they are intelligent, conscious, …","tags":["cognitive science","consciousness","llms"],"title":"LLMs and Theories of Consciousness"},{"body":" Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it out.) Note that I go into a little more statistical detail here, although most of the below is readable by non-data scientists.\nA/B testing, or on-line controlled experimentation, is conceptually simple. Show half of your users a new experience, collect some data about their behavior, look at the data, then decide what to do. It can get tricky in the implementation details, but I want to focus here on the decision. The reason we do A/B tests is that we don’t yet know enough what the effect of this new experience will be, on user behavior and on our metrics, for us to make a decision. We’re uncertain. We need to reduce our uncertainty about the impact of the new experience, so that we can make a decision. The experiment will, if we’ve planned it right, give us enough confidence to make the right decision.\nOK, but not all decisions are equal. Some will drastically affect users, cost real money, require teams of engineers to implement and support. Should you implement a deep-learning-based recommender system? Sounds expensive. You’d better know a lot about the impact of the change. Others are trivial. Should this button be green or blue? It doesn’t seem like the decision-making should be the same in both cases.\nA/B testing, I think, should be thought of as a toolbox, a collection of approaches that are useful in different situations. Those approaches, the tools in the toolbox, are not just the statistical tests. They also include the process, including critical communication patterns.\nAs a data scientist supporting A/B tests, I’m constantly talking with stakeholders to get agreement on key aspects of each test – why we’re running the test, how we should measure success, and what thresholds we’ll use. Then, after I figure out how long we need to collect data, we go ahead and collect the data, analyze it, and, together, make a decision. Each tool, each type of decision, needs different approaches for these discussions.\nBenn Stancil, co-founder of Mode, has argued that the most important goal for data people is to speed up decision-making in organizations. For A/B testing, there are lots of things we can do to speed up the process, most of them up front. Having clear, standardized ways to communicate can avoid surprises, reduce disagreements and let the team move on to the next thing promptly. And in some cases, understanding the motivation behind a test may even allow you to speed up the test itself.\nIn my experience, these 5 types of decisions cover most A/B testing scenarios.\nSuperiority Bias-to-Ship Agnostic Measurement Lean Startup Each decision is appropriate for a different scenario, a different motivation for running the experiment. Once everyone is aligned on why we’re running a test, with this framework, most of the rest of the process becomes details.\nSo let’s go through these decisions. In each case, I’m assuming you’re testing a New Experience versus the Control Experience, and have a single Key Metric that you care about that may be affected by this experience. To make it easy to follow, I describe when each decision is used, appropriate scenarios for this decision, how to plan for the experiment, and how to actually make the decision, once the data is in. Then, I provide some technical notes about the statistical approaches used in the planning and decision-making.\nFigure: Perhaps-useful intuitions for the five types of decisions.\nSuperiority Decision Used When: You only want to roll out the New Experience if it is almost certainly better than the Control. Appropriate for: Changes that would be costly to fully implement and maintain, that would make your systems more complex. It’s a conservative test! Planning: Run the test until you’re quite likely to detect the smallest positive impact on the key metric that would cause a roll-out decision. Decision: Statistics say the New Experience is almost certainly better. These are intuitions that data scientists, product managers, and others can share. We can agree, before a test, that this is the type of decision we’re making. Then, we can agree on particular quantitative thresholds for the italicized phrases.\nFor Superiority decisions, reasonable numbers might be: almost certainly = 95%, quite likely = 80%, smallest positive impact = 1%.\nAs I discussed previously, for this decision, this ends up being essentially the same process as classical null-hypothesis statistical testing. You can use standard statistical methods to determine power, the length of time required to get enough data. Then, when you get to the post-experiment analysis phase, you can use the phrases from above again. Instead of confusing mathy jargon like p \u0026lt; 0.05, we can instead say “It is almost certain that the New Experience is Better than the control.” That’s what we agreed we needed to make a decision!\nAnd when talking about the size of the impact, we can use phrases like “quite likely”, representing statistical uncertainty intervals, to help stakeholders understand the amount of uncertainty there is remaining. “The New Experience is quite likely between 0.3% and 0.7% better.” We’ve run the test to reduce our uncertainty, and have made a decision according to pre-determined criteria. Fantastic – let’s move on to the next test!\nBias-to-Ship Decision Used When: We want to roll out the New Experience unless it is plausibly substantially worse than the Control. Appropriate for: Changes that you want to make for non-metric reasons. A refactor or introduction of new technology that’ll be easier to maintain. A rebrand. Planning: Run the test until you’re quite likely able to rule out a bad outcome. Decision: Roll out unless statistics say the New Experience is plausibly substantially worse. Note that failure probably means “revise and retest”, not “abandon”! For a Bias to Ship experiment, you want to be pretty sure a substantial negative impact won’t happen when you fully roll out the change. It turns out that the math for classical A/B testing is almost exactly symmetrical, and you can use the same approach as a Superiority decision, and just flip the sign. So instead of the smallest positive impact being 1%, the largest negative impact you can live with is -1%. Everything else is the same.\nThis approach is what Georgi Georgiev of the site Analytics Toolkit calls an “Inferiority Design”. It’s a commonly used technique particularly in medical statistics, but names for things matter, so I’d recommend calling this “Bias-to-Ship” instead.\nAgnostic Decision Used When: You want to decide quickly while likely making the correct decision. Appropriate for: Cosmetic, no-cost decisions. Should the button be blue or green? Planning: Run until you’re likely to make the correct decisions, assuming you run many similar tests and have a sense of the range of potential impacts. Decision: Go with whichever variant is winning, by any amount. (!) For an Agnostic experiment, the approach is a little different, and no longer resembles the NHST framework. Instead, I think a Bayesian approach makes sense. If you have been A/B testing changes to your web site for a while, you probably have a sense of the distribution of effect sizes that changes make. Most of the time they probably have little effect on your key metric, but occasionally a change is measurable and important. Your estimate of that is the prior distribution on the effect.\nFor the low-stakes tests in an agnostic experiment, you may not need to run with the same amount of power as you would for a Superiority decision – being reasonably confident (“likely” vs. “very likely”) is adequate. You also don’t care about the magnitude of the effect, you just want to be pretty sure you’re making the right decision (i.e., the estimated sign of the effect is correct, see Gelman \u0026amp; Carlin, 2014 and Inmbens, 2021).\nBy simulating many experiments, with effect sizes drawn from the prior distribution, and a range of experiment durations, you can identify how long you’d need to run the experiments to get most of the decisions (say, 80%) right. Then, there’s no need for a statistical test to be run at all once the experiment ends – you’ve already collected enough data to be adequately certain that the sign of the impact that you measured is on the correct side of zero. Very quick and easy decision!\nMeasurement “Decision” Used When: You aren’t making a decision – you already know you definitely will or will not make a change, but you want to be relatively precise in your estimate of the impact from the New Experience. Useful for learning, for guiding future designs, and for understanding the impact of past, untested changes. Planning: Run until you have the amount of uncertainty you want. E.g., relatively precise = ±2% Decision: N/A For a Measurement quasi-decision, the statistical process may be similar to a Superiority design, but the communication is quite different. Saying “I want a measurement of the effect accurate within + or - 1%” is not that different from saying “I want to be able to detect a change of 1%.” (But note that this is a two-sided statement, while a Superiority design is usually one-sided.)\nLean Startup Decision Used When: You only want to roll out the New Experience if it is almost certainly MUCH BETTER than the Control. MUCH BETTER would change your organization and product direction dramatically, e.g. a 50% improvement in conversion! Appropriate when: Small impact is meaningless and should be ignored, as if you’re a startup trying to find product-market fit. You have many ideas, most won’t work, but one might be game-changing, and the only thing that matters is if you’ll find it or not. Decision: If you see a huge impact, it turns out that it’s more likely from a game changer vs. from noise! A Lean Startup decision is a super-aggressive variation on a Superiority Decision. The statistics are the same as a Superiority design, but during the power analysis phase, the test is set up to be deliberately underpowered. Instead of the smallest positive impact that would lead to a roll-out, you only need enough power to detect an enormous, game-changing effect.\nThe intuition is that, of all the things you could be doing on your web site, most will have minimal impact, but a few of them might have a game-changing impact. Randomness, however, is less likely to cause such a big impact. (Formally, the distribution of impacts from changes is probably fat-tailed, compared to the normal distribution of random sampling error. See Azevedo et al., 2020 for a detailed analysis.) So if you do measure an unexpectedly big win, it’s probably real. Given this observation, you can run more high-risk high-reward experiments in less time, and hopefully find a game-changer.\nSo those are the five types of A/B test decisions. I have now introduced this framework at two different companies. In both cases, I found that the naming of these five types of decisions dramatically streamlined test setup and decision-making. If you take the time to implement something like this at your organization, please reach out! I’d love to hear how it went!\n","date":1688688e3,"id":"25","link":"/2023/07/the-five-types-of-a-b-test-decisions/","section":"post","summary":"Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an …","tags":["a/b testing","statistics","data science"],"title":"The Five Types of A/B Test Decisions"},{"body":"Recently, tech-journalism site The Markup ran a long, detailed, critical investigation of a predictive machine learning model used by the State of Wisconsin to identify public school students at risk of not graduating. I mostly agree with the conclusions of the piece -- the system appears not to be fit for purpose and needs to be substantially improved -- but I want to comment on several aspects of the model and the Markup’s reporting. Although I know nothing about the Wisconsin model beyond what is reported, I know a lot about predictive student success risk models, having led a team of data scientists who built related models used by colleges and universities when I worked at EAB from 2014 through 2016.\nThe Markup describes the Dropout Early Warning Systems (DEWS), a system that, for each Wisconsin 6th through 9th grader, predicts whether the student is likely to graduate from high school on time. Through deep reporting and access to internal data, they were able to learn a great deal about how the system works, and importantly, how well it works -- or doesn’t -- as a whole, and for racial and ethnic minorities.\nTheir conclusions include \u0026quot;DEWS is wrong nearly three quarters of the time when it predicts a student won’t graduate on time\u0026quot; and \u0026quot;it’s wrong at significantly greater rates for Black and Hispanic students than it is for White students.\u0026quot;\nThe three points I want to make are:\nIt’s worthwhile to build predictive models of student success as part of advising and support systems. Getting these models to work well for racial and ethnic minorities is i mportant but difficult work, and it seems like the DEWS model got it wrong. Language and design matter, and both DEWS and The Markup seem to have confused notions of \u0026quot;accuracy\u0026quot; and \u0026quot;risk\u0026quot;, and how to communicate about model predictions. Why build these models in the first place? Because of limited resources. In a perfect world, every student would have enough attention from expert advisors to support them at every step in their educational journey, getting them the help and resources that they need, so that they are engaged, learn successfully, and graduate on time. Both in K-12 and higher education, though, budgets do not allow this sort of bespoke advising team, and stretched-thin advising staff must do their best to support students who voluntarily seek help, while also doing limited outreach to other students who could most benefit from their support. But who will most benefit? This is where models can help, by triaging students, identifying those who should get a deeper look, and perhaps identifying useful avenues for outreach.\nThe Markup describes how the origin of the DEWS model was to support efforts to improve graduation rates by racial and ethnic minorities in Wisconsin, an important goal, and potentially a cost-effective way to do so. As they report, \u0026quot;94 percent of White students graduated on time last year, [but] only 82 percent of Hispanic and 71 percent of Black students completed high school in four years.\u0026quot; Something needed to change. However, it’s clear that the resulting model, and the system it is part of, is not helping the state build an equitable education system. Hopefully the attention the DEWS system is getting now, as a result of this reporting, will lead to improvements, and soon.\nMuch of the article describes how the system is less accurate for racial and ethnic minorities. They include a quote from a state employee involved in the system: \u0026quot;the model... over-identifies Black, Hispanic and other students of color among the non-on-time graduates.\u0026quot; That is, the model is mis-calibrated. Ideally, if this model were to say that 100 students are each 75% likely to graduate on time, then around 75 of those students should do so. In this case, it seems like the actual graduation rate for minority students with this risk score is higher than 75%, so those students are getting inappropriately flagged.\nWith models of this sort, it’s tempting to say \u0026quot;don’t use race or ethnicity as a predictor in the model\u0026quot;. The Markup quotes a professor who says \u0026quot;...[t]hey had demographic factors as predictors and that’s going to overemphasize the meaning of those variables and cause this kind of effect.\u0026quot; But ignoring race, ethnicity, or other correlated factors doesn’t make those factors, or the history of white supremacy, go away. In fact, it’s sometimes better to use those factors, but carefully.\nThe problem arises when the model is too simple, if it treats factors as information that gets added together, along the lines of those \u0026quot;you might be at risk for a heart attack: 3 points if you’re over 60, 2 points if you have high blood pressure, etc.\u0026quot; simple models you sometimes see in magazines. Learning from historical data, it’s easy for the model to improperly conclude that being non-white reduces your likelihood of graduation, yielding scores that are too low. A better approach is to identify other factors that may be differentially important based on a student's race or ethnicity, and let the model estimate those \u0026quot;interaction\u0026quot; effects. In higher ed, for instance, it’s important to look at SAT or ACT scores in conjunction with race, because of known issues with those tests. Equally skilled minority students often get lower scores because of factors such as limited access to expensive prep classes. Everything else being equal, in some cases, a minority student with the same SAT score as a white student might be more likely to graduate!\nIncluding race and ethnicity actually makes the models better, if you do so the right way, leveraging knowledge of how the world works and what the data means. A race-aware algorithm might treat SAT or ACT scores as less informative and predictive for non-white students, or might apply a correction to counteract the known bias. It’s pretty clear, from The Markup’s reporting, that DEWS and the State of Wisconsin did not try to do this.\nThe Markup repeatedly makes statements such as \u0026quot;[DEWS] was wrong nearly three quarters of the time it predicted a student wouldn’t graduate on time,\u0026quot; implying that the model is fundamentally flawed. But this issue is not about the model at all, it’s about the language used to describe the predictions.\nThe predictive model is only part of a larger system that also includes the human administrators who use the model’s scores to take action, or not. And humans aren’t great at interpreting numerical scores. So, a reasonable decision was made to discretize the 0-100 probability score (which, as described above, is well-calibrated if, when you take all of the students with a score of 80, 80% of them later graduate) into red/yellow/green risk categories, where in this case the red/yellow boundary is at 78.5. Given that most students in Wisconsin graduate, it's probably reasonable to put a category boundary there, but only if the categories are described appropriately.\nIf you take a student with a score of 75 (indicating that you estimate that of 100 very similar students, 75 will graduate), and give them a \u0026quot;red\u0026quot; score, it’s critical that you not call this \u0026quot;high risk\u0026quot;, even if the student is in the riskiest group. Anybody unfamiliar with the technical details will assume that \u0026quot;high risk\u0026quot; means \u0026quot;very likely will not graduate,\u0026quot; which is incorrect in this case. Instead, all of the messaging around the model should be carefully designed to match peoples’ intuitive understanding of qualitative terms. A score of 75 means that the student will likely graduate!\nAdditionally, it’s important to avoid self-fulfilling prophecies, where the language used suggests that nothing can be done to improve a student’s odds or outcome. It’s too easy to interpret phrases like \u0026quot;high risk\u0026quot; as being an essential and unchanging part of a student’s being, not something that can and will change over time, or that could be changed with effort. Ideally, the model should be identifying students who would benefit from additional support, and describing them as such. I would recommend, in this case, using terms such as \u0026quot;needs support\u0026quot; or \u0026quot;investigate\u0026quot; instead of \u0026quot;high risk\u0026quot;. These terms should be used consistently in the system’s user interface, in training, and elsewhere.\nAnd, if the threshold is indeed at 78.5, maybe red isn’t the right color? Red implies \u0026quot;something is on fire\u0026quot;, but a student who is currently 75% likely to graduate maybe just be yellow? Or maybe use an emoji like 👀 instead of a color, to indicate that it’s worth watching and supporting this student closely? Design decisions like this can drastically affect how the system is used, and thought and user research should go into ensuring that the users of a machine learning system work properly with it.\nIt’s clear from screenshots in the article that the authors of the DEWS tool missed an opportunity for good design, and so administrators and educators were mis-interpreting the results. The Markup reports a number of insightful comments that confirm this. It’s a shame that the system’s language is misleading, but the good news is that this sort of thing is relatively easy to fix, even without changes to the predictive model itself.\nIt’s also regrettable that the journalists at The Markup mis-understood this point, as it takes away from their strong critique of the model itself. The big issues with the predictive model are that it’s not very predictive -- it seems to mostly provide scores in the relatively narrow range of 75 to 95 -- and that for critical subgroups such as racial and ethnic minorities, its predictions are often biased high or low. Treating the design issue as a modeling issue confuses the matter, and misses an opportunity to discuss the importance of design to the system’s success. (The Markup does appropriately discuss how inadequate the training was, which is another common problem with this sort of system.)\nAgain, I applaud The Markup’s deep investigation into an important issue, and for that matter, I applaud Wisconsin’s efforts to attempt to address educational inequalities. Hopefully, student success models and systems in Wisconsin and elsewhere will continue to improve, allowing more students, especially those from disadvantaged backgrounds, to benefit from a strong education.\n","date":1685318400,"id":"26","link":"/2023/05/a-critique-of-the-markup-s-investigation-into-predictive-models-of-student-success/","section":"post","summary":"Recently, tech-journalism site The Markup ran a long, detailed, critical investigation of a predictive machine learning model used by the State of Wisconsin to identify public school students at risk …","tags":["machine learning","journalism","predictive"],"title":"A Critique of The Markup’s Investigation into Predictive Models of Student Success"},{"body":"The \u0026quot;best practice\u0026quot;, when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent 2016 statement that \u0026quot;...business... decisions should not be based only on whether a p-value passes a specific threshold.\u0026quot; Wait, what? Are we making bad decisions from A/B tests? Should we stop using p values and do something else?\nMy answers are yes -- we are too often making bad decisions, and sometimes -- p values are only useful for certain types of A/B test decisions. In this post, I'll talk about the specific decision pattern where p values remain a reasonable approach, taking the somewhat contrarian position of defending them against the ASA's guidance (while still almost entirely agreeing with everything else their statement says).\nWhy p Values Are Bad Let's back up. If you took one of those Introduction to Statistics classes, you probably were taught a few things about \u0026quot;statistical significance\u0026quot;, \u0026quot;p values\u0026quot;, and \u0026quot;null hypotheses.\u0026quot; You probably know that before you run an experiment, you do a \u0026quot;power analysis\u0026quot; to figure out how long you need to collect data for. Then, after you run the experiment, you can plug the results into a formula to compute a value called p, and if that number is less than 0.05, then the experiment was \u0026quot;statistically significant\u0026quot;, otherwise you cannot exclude the null hypothesis. You might also know that this process was developed in the first half of the 20th century to support academic science, especially agricultural and medical research. The framework is called Null Hypothesis Statistical Testing (NHST), and it is ubiquitous.\nBut you might not know how much pushback has happened in recent years. In 2016, the American Statistical Association, the professional society for statisticians in the US, published a statement on p values. Here is the summary (bold added for the most important two points for our purposes):\nP-values can indicate how incompatible the data are with a specified statistical model. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. Proper inference requires full reporting and transparency. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. If you're like most people I've worked with, marketers or product managers or analysts, the above comes as a shock. Here we are, trying to make important decisions about our business, and the experts are specifically saying that our approach is bad!\nWhy did the ASA say this? It's not because the p value is being incorrectly calculated, it's because in many cases it's doesn't mean anything relevant to the decision that's being made. What it does mean is the following somewhat convoluted statement: it's the probability, if the experiment were re-run again exactly the same way, and if in fact there was zero impact of the experimental manipulation, that the results would be as large or larger than what was observed.\nIt's not clear at all how this confusing probability crossing an arbitrary threshold is anywhere near the right way to decide whether to make a change to your web site or app, or not.\nDecision Making in Science and Industry There seems an interesting tension between the needs of practitioners in modern organizations, and the problem that p values were originally intended to solve. Put simply, a low p value, if you ran the experiment properly, means that probably something interesting is going on. In science, that’s a great start -- it begins a conversation that will take years to resolve, using replication, formal models of mechanisms, and other methods of converging evidence. It's also worth noting that in science, the goal is often to estimate the impact of a change, or of some parameter of the model. p values only let you, best case, make an argument that an effect size is non-zero. So yes, for science, treating p \u0026lt; 0.05, on its own, as anything like conclusive or even particularly useful, is inappropriate.\nBut in industry, we need to have conversations about what data will mean for us first, then decide as quickly as we can what to do once we have data. Taking years, or even weeks, to decide what to do with the results of an experiment is not acceptable in almost any business. We need to take the results, then use agreed-upon decision rules and communication patterns to move forward.\nSo, can we find appropriate ways to use statistical tools to fit into the decisions and the communication patterns that we need to have? I think we can. And in one particular type of A/B test, the NHST approach of a p value crossing a threshold may even be the right tool for the job.\nSuperiority Decisions Sometimes, when you're running an A/B test, you only want to roll out a new experience if it is almost certainly better than the existing experience. This is a conservative decision, appropriate when the thing you're testing is more expensive to maintain (or to fully build) than the control experience. Perhaps you're adding a fancy and expensive recommendation module to a page, or you built a quick-and-dirty version of a feature that won't scale, just for the test. This may not be the most common type of A/B test that people run, but it certainly happens. I like to call this a \u0026quot;Superiority Decision\u0026quot; test.\nBefore the experiment runs, you need to figure out, and as a team agree upon, two things -- how long to run the experiment for, and how you'll make a decision based on the results you see. Before jumping into statistics, let's frame these qualitatively:\nWe want to make a Superiority Decision. We want to run the experiment until we're quite likely to detect the smallest positive impact that would cause a rollout decision. Then, after we've collected the data, we will roll out the change if statistics say that the treatment experience is almost certainly better than the control experience.\nI've found it extremely valuable to start with statements like this, getting agreement among stakeholders, before the experiment begins. In particular, the smallest positive impact value requires review of past experiments, discussion of cost-benefit tradeoffs, and a key decision to be made.\nTranslating Business Decisions to Statistics The statement above is essentially the classical NHST approach. For instance you could set quite likely to 80%, smallest positive impact to 1% or whatever seems appropriate for the experiment, and almost certainly to 95%. By agreeing on this, you can then use standard power analysis techniques to determine how long to run the experiment, then use p \u0026lt; 0.05 as the decision criteria.\n(As I discussed previously, although in general p \u0026lt; 0.05 does not mean \u0026quot;95% likely to be better\u0026quot;, in the very simple case of a t test for a properly-executed A/B test with adequate power, it's very close, and certainly adequate to mean \u0026quot;almost certainly better\u0026quot;.)\nWhen I've used this approach, I've paired this setup with standardized language around the decision, such as:\nIt is almost certain that the New Experience is better than the Control. The New Experience is quite likely between 0.3% and 0.7% better.\nThe first sentence is based on the p value being lower than the 0.05 threshold, but notably, I don't actually report the p value itself in this summary. Instead, I report uncertainty intervals around the expected change, as discussed previously. This avoids all of the confusion around what p means, and avoids anchoring on point estimates or other numbers, but behind the scenes, relies on the NHST pattern.\nOn the ASA Statement Returning to the ASA statement, here are my comments on each of their assertions:\nP-values can indicate how incompatible the data are with a specified statistical model. Agreed.\nP-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. For very simple cases, which is what A/B tests are, p values below a threshold can be interpreted as an indication that we have knowledge that should allow us to make a specific decision. Is this \u0026quot;true\u0026quot;? Maybe not, but it's still useful.\nScientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. In some cases, such as the Superiority Decision described here, business decisions should be made expediently based on pre-determined rules, which can be p values below a threshold value. But in many other cases, other decision criteria should be used.\nProper inference requires full reporting and transparency. Yes, but proper reporting to non-statistically-fluent stakeholders also requires not misleading them or burying them in jargon. As data scientists in industry, we are hired for our ability to guide the business using the tools of statistics. We should be intellectually honest without being offputting. Starting with qualitative statements that everyone can agree on goes a long way in the right direction.\nA p-value, or statistical significance, does not measure the size of an effect or the importance of a result. Completely true. If you have to mention p values in a report, it's much better to say p \u0026lt; 0.05 than p = 0.0034 or whatever, which will cause stakeholders to say wince-worthy things like \u0026quot;very statistically significant.\u0026quot; The misnomer \u0026quot;statistical significance\u0026quot;\u0026quot; just means that you've crossed a threshold that leads to a particular decision. You've pre-determined the importance of the result, and you explicitly do not care, for a Superiority Decision, how big the effect size is.\nBy itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. Right, but it can be adequate for making a decision, specifically a Superiority Decision. It's not a measure of evidence, just a criterion that lets us take our win (or our loss) and move on.\n","date":1682985600,"id":"27","link":"/2023/05/p-values-are-useful-for-a-b-tests-sometimes/","section":"post","summary":"The \u0026quot;best practice\u0026quot;, when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of …","tags":["a/b testing","statistics","data science"],"title":"p Values Are Useful for A/B Tests, Sometimes"},{"body":" A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various sorts – say, the proportion of people who visit your web site who buy at least one pair of shoes. Although the underlying data is just four numbers (visitors and purchasers, from the control group and the test group), there are lots of ways to compute statistics and present those numbers. Not all of those ways are ideal for supporting communication.\nIn this post, I’m going to walk through my favorite way of presenting A/B test results for conversion rates – as ratios, with uncertainty intervals around those ratios. For instance, you might be able to say something like “this test very likely increased the conversion rate by somewhere between +0.6% and +4.8%, perhaps around +3%.” (That would be a really good result for almost any change on any web site, by the way!) Business people think in percents, all day, every day – almost every KPI is presented as quarterly or annual growth. So presenting this way means you’re speaking their language and will get their attention.\nBut, that’s not how most web sites and A/B testing tools present the numbers. Suppose you had 100,000 visitors to your A and B variants of your site, 15,000 of them converted in the A (control) variant (15%), and 15,400 converted in the B (test) variant. Let’s look at how some of those tools might present those results.\nAt AB Testguide, a web calculator you might get when you type “A/B test calculator” into a web search engine, you get the 2.7% relative ratio (\\(\\frac{15400-15000}{15000}\\)), but everything else is (in my view) a confusing mess:\nFigure 1: Results from AB Testguide web site.\nThe bottom-line box up top is nice, but it shows unlabeled uncertainty intervals, so you can’t really tell how uncertain each conversion rate is. (As an aside, I often follow Gelman’s advice to use “uncertainty” instead of “confidence”, but it’s the same thing.) Those normal curves are hard to read and take up a ton of space. They’re supposed to communicate that each measurement has uncertainty, and that there’s not much overlap, but it’s unclear what “not much” means, or how you’re supposed to use the graph to make a decision. The boxes below show the steps in the standard frequentist computation of the \\(p\\)-value for the ratio test, which is what the app is doing. That’s fine, I guess, but none of these numbers help you much with communication. And if you’re using an app like this, and not, say, the tools built in to R or Python, you’re probably not a statistician, so this is a little bit of blinding you with science. Speaking of R, let’s look at how its built-in prop.test function presents the results. This is a standard tool for statisticians. It’s one line of code from a standard library:\nprop.test(c(15400, 15000), c(100000, 100000)) ## ## 2-sample test for equality of proportions with continuity correction ## ## data: c(15400, 15000) out of c(1e+05, 1e+05) ## X-squared = 6.1756, df = 1, p-value = 0.01295 ## alternative hypothesis: two.sided ## 95 percent confidence interval: ## 0.0008431498 0.0071568502 ## sample estimates: ## prop 1 prop 2 ## 0.154 0.150 This output gives you what you’d need to format your results for an academic paper. You might say: “The test increased the conversion rate significantly (\\(\\Chi^2(1)=6.18, p=0.01\\)).” That’s not really going to help you talk to a product manager. It does give a confidence interval of [0.0008 to 0.0071], but that’s for the absolute difference of the proportions of 0.004, not the relative change. Again, you’re better off if you can consistently communicate using relative measures.\nWhat about commercial products? Lots of tech companies use Optimizely, which has nice tools for running A/B tests from either the front-end, where marketers can change the UI without software engineers, or the back-end, where engineers can run robust tests that aren’t invalidated by ad-blockers. But I don’t love their presentation of results, such as for this A/B/C (two tests vs. one control) test from their documentation:\nFigure 2: Example of Optimizely’s user interface on test results.\nThey do a nice job of showing the number of visitors, the conversion rate, and the relative improvement (in this case, negative). But they show a difficult-to-interpret unlabeled confidence interval which just says whether the interval includes 0 (no change) or not. And they use an unusual presentation of \\(1-p\\) for statistical significance. This is likely to draw non-statisticians to misinterpret this value as “the likelihood that the improvement number (e.g., -0.73%) is correct.” The correct meaning is more like “the likelihood that a re-run of this experiment would not show an effect of this size or larger, if the true effect size is exactly 0.” So it’s not supporting effective or correct statistical communication. So what might be better? I think that focusing on the relative ratio (as that web site and Optimizely do) is a good idea, but we should lean into uncertainty intervals more. In particular, instead of trying to show uncertainty intervals on both of the raw measurements (as the web site does), we should compute uncertainty intervals on the ratio, and focus on that interval even more than the point estimate of the ratio.\nFor the toy example from above, I’d recommend saying something like: “the improvement is very likely between 0.6% and 4.8%, with a point estimate around 3%.” If you wanted, you could draw a little graph of this, and I’ll show you a decent way to do that below, but really, the three numbers are enough.\nBut where did those numbers come from? As it turns out, the ratio of two proportions (each of which is an observation from an underlying binomial distribution) has an easy-to-compute distribution. The R code below computes the 95th percentile range of that distribution, representing the uncertainty interval. (The “very likely” above is a bit of hand-waving to communicate this to a non-statistician. More on that below.)\ncr_ratio \u0026lt;- function(a_ct, b_ct, a_conv, b_conv, percentiles=c(0.025, .5, 0.975), as_percents = TRUE) { a_prop = a_conv / a_ct b_prop = b_conv / b_ct m = log(b_prop / a_prop) v = ((1 / a_prop) - 1) / a_ct + ((1 / b_prop) - 1) / b_ct ret = exp(qnorm(percentiles, mean=m, sd=sqrt(v))) if (as_percents) ret \u0026lt;- 100 * (ret - 1) names(ret) = percentiles * 100 ret } cr_ratio(100000, 100000, 15000, 15400) ## 2.5 50 97.5 ## 0.5627688 2.6666667 4.8145807 Removing insignificant digits and rounding appropriately, this yields the 0.6% to 4.8% range and the point estimate.\nYou can, with a bit of effort, create visualizations that get at the most important things to communicate: the uncertainty interval, and the “no impact” baseline:\ndf \u0026lt;- as_tibble(t(cr_ratio(100000, 100000, 15000, 15400, as_percents=FALSE))) names(df) \u0026lt;- c(\u0026#39;min\u0026#39;, \u0026#39;pt\u0026#39;, \u0026#39;max\u0026#39;) df$x \u0026lt;- 1 labels \u0026lt;- tribble( ~x, ~y, ~txt, \u0026quot;purchase rate\u0026quot;, df$min, scales::percent(df$min-1, .1), \u0026quot;purchase rate\u0026quot;, df$max, scales::percent(df$max-1, .1) ) ggplot(df, aes(x, ymin=min, y=pt, ymax=max)) + geom_crossbar(width=.2) + geom_text(data=labels, mapping=aes(x=x, y=y, ymin=NULL, ymax=NULL, label=txt), nudge_x=.2) + geom_hline(yintercept=1, color=\u0026#39;blue\u0026#39;) + scale_x_discrete(\u0026quot;\u0026quot;) + scale_y_continuous(\u0026quot;B/A Ratio\u0026quot;, labels=scales::percent, limits=c(.92, 1.08)) + coord_flip() + theme_minimal() This display is handy to standardize, as you can show multiple different conversion rates on the same graph, giving clear context to the results of the experiment.\nBefore certain statisticians in the audience write angry comments, I’m well aware that interpreting frequentist intervals correctly requires some hair-splitting language, and that Bayesian credible intervals are more directly tied to peoples’ intuitions. As it turns out, though, in practice at the scale that most web A/B tests run, there’s no practical difference. You can interpret frequentist intervals such as the above as if they were Bayesian, and describe them the “wrong way”, as if they were saying something about the beliefs you should have about the true values. Here’s an illustration, generating the Bayesian credible interval on the posterior distribution:\n#plotBeta(15,85) # we think CR is around 15% AB1 \u0026lt;- bayesAB::bayesTest( c(rep.int(1, 15400), rep.int(0, 100000-15400)), c(rep.int(1, 15000), rep.int(0, 100000-15000)), priors = c(\u0026#39;alpha\u0026#39; = 15, \u0026#39;beta\u0026#39; = 85), n_samples = 1e5, distribution = \u0026#39;bernoulli\u0026#39;) round(summary(AB1, credInt=rep(0.95, length(AB1$posteriors)))$interval$Probability * 100, 3) ## 2.5% 97.5% ## 0.562 4.820 Because the number of observations is high, and our prior is relatively broad, this is essentially the same interval as we computed above. So we’re justified in the hand-waving that makes communications with business people much, much clearer. We can defensibly say, if pressed, that “very likely” is equivalent to “a range covering 95% of the credible values of the true ratio.” Plus, the closed-form frequentist approach is much, much faster than the numerical simulation required to compute the Bayesian intervals.\nTo sum up:\nFor A/B tests in industry, you should speak business language, and talk about the relative improvement from the changes you’re testing. But, you need to be honest about your uncertainty. Lead with the uncertainty interval, and mention the point estimate only secondarily. Use the closed-form computation above to compute the uncertainty interval. Even though Bayesian computation is better in principle, it won’t help you help your organization make better decisions, it requires more work on your end, and it’s likely to confuse people. If you end up using this approach to communicating A/B test results, I’d love to hear about your experiences! For me, I’ve found that consistently communicating this way led to easy conversations and fast decision-making.\n","date":1660953600,"id":"28","link":"/2022/08/communicating-a-b-test-results-for-conversion-rates-with-ratios-and-uncertainty-intervals/","section":"post","summary":"A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have …","tags":["a/b testing","data science","statistics"],"title":"Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals"},{"body":"I recently read Will Larson's excellent book Staff Engineer: Leadership beyond the management track. Larson covers the individual contributor (IC, not management) roles that software engineers fill after they are promoted past Senior Software Engineer, with titles like Staff and Principal (\u0026quot;Staff-plus\u0026quot;). In the book, he synthesizes his own experience and the experiences of a number of other Staff-plus engineers, and provides great insights into how to get promoted to that level, and how to succeed at it. Great book -- you should read it.\nAs someone who has had both management roles and similar IC roles, I thought I would comment on a few things from Larson's book, and highlight a few things that jumped out as particularly resonant. These comments are based mostly on my recent experience as Staff Data Scientist (mostly focused on Search) at Teachers Pay Teachers, and some initial thoughts as Principal Data Scientist (mostly focused on Growth) at Algolia, where I've been for the last month. (I'm going to talk primarily about insights-focused data scientists, what others have called Type A (Analysis). Data Scientists who mostly build machine learning systems have a very different role at this level.)\nQuotes are of Larson, unless otherwise specified.\nArchetypes (p.12) The more folks I spoke with about the role of Staff-plus engineers at their company, the better their experiences began to cluster into four distinct patterns...\nThe Tech Lead guides the approach and execution of a particular team...\nThe Architect is responsible for the direction, quality, and approach within a critical area...\nThe Solver digs deep into arbitrarily complex problems and finds an appropriate path forward...\nThe Right Hand extends an executive's attention, borrowing their scope and authority to operate particularly complex organization.\nAs a data scientist, and particularly one who used clustering to define subtypes of Data Scientists in the past, this warmed my heart. The types of work that I've done roll up well to versions of these clusters. At TPT, I was a co-lead of the Search team, working with a Product Manager and an Engineering Manager to guide the approach of that team. I wasn't the Tech Lead -- someone else had that formal role -- but I very much helped that team be grounded in data and user experiences with search, and in some sense split responsibilities with the PM.\nOther projects rolled up to other patterns. I worked closely with the Analytics team on A/B testing, and helped TPT get stronger and smarter about how they ran tests. In some sense, I was the \u0026quot;Architect\u0026quot; of A/B testing, although that sounds more formal than it was. I also did some special projects in partnership with other insights teams, such as a causal statistical analysis of a major product launch, that feel like a \u0026quot;Solver\u0026quot; role.\nI haven't really had a Right Hand role, but can imagine how a data scientist could work closely with a CEO. One part of a CEO's role, I've heard described, is to interpret external reality to get the organization aligned on why they're doing what they do. Interpreting reality is very much what data scientists do too.\nWhat Staff-plus Data Scientists Do (p. 20) Staff engineers do [the same tasks as Senior engineers], but whereas previously they were the core of their work, now they're auxiliary tasks. Their daily schedule [is primarily now]: setting and editing technical direction, providing sponsorship and mentorship, injecting engineering context into organizational decisions, exploration, and what Tanya Reilly calls being glue.\nYes, this. The primary thing I want to highlight is what successfully injecting data science context into organizational decisions means. A thing that data scientists are particularly good at is reasoning about uncertainty. What we can do to support organizational decision-making is to help leaders and management understand what is more certain and less certain, to find ways to reduce uncertainty, and to make the best decisions possible. Tools for this include A/B testing, good data visualization, statistical modeling, stochastic simulation, and other tools from the data science toolkit.\n(p. 57) [As] you look at how software changes over time, there are a small handful of places where extra investment preserves quality over time... I call those quality leverage points, and the three most impactful points are interfaces, stateful systems, and data models...\nData models [constrain] your stateful system's capabilities down to what your application considers legal... A good data model is tolerant of evolution over time. Effective data models are not even slightly clever.\nOoh, data! For software engineers, data models are how the software represents the world that the organization and the software's users care about. It's critical to get right. For data scientists, we also care about representing the world, and we can be an important voice in ensuring that the data we use and the work that we (and analysts, and business intelligence engineers, etc.) do actually approximates the real world well. At the staff-plus level, we should have both the business context and the technical skills to help the organization be better at measuring, finding insights, and making decisions that reflect the complexity of the world, while working around problems in the systems that provide our data.\n(p. 192) [From an interview with Katie Sylor-Miller at Etsy]\nI've definitely found that... it's taking longer to actually find the dedicated focus time to write code as my calendar fills up with meetings... I'm much more focused on identifying areas for opportunity and then trying to sell that as work that my team or other teams should be doing.\nIn classic management texts, managers are urged to find the highest-leverage thing that they can do, and delegate the rest. The same applies to Staff-plus ICs. Although I spend a lot less time in meetings than I did when I was a manager, I spend more time writing documentation and processes, or helping others up-skill, or researching problems and proposing projects.\n(p. 195) [More from Katie Sylor-Miller]\nI also think that Staff Engineers should have a broad understanding of all of the adjacent fields of work to their own specialty. For me, working in the frontend, I put a lot of time and effort into understanding marketing, business goals, user experience, visual design, the view and business logic layers on the server, how we ship code to the browser, [etc.] Having expertise in all of these different areas makes it easier for me to see the broader impact of my technical decisions and understand those tradeoffs better.\nYes, and related -- I tend to think that an effective data scientist working in the product organization at a tech company should be able to operate as a software engineer a couple of levels lower. So if a Staff Data Scientist is a \u0026quot;level 5\u0026quot; role (which it often is), then someone filling that gap should be able to operate as a Senior (level 3) Software Engineer. This doesn't mean that you should be spending your time writing production code (not very high leverage), but it does mean that you should be able to read production code and be able to think about and communicate about software architecture and implementation at that level. (Of course, if you work in another type of organization, or outside of product, other rules of thumb probably apply.)\nHow Staff-plus Data Scientists Succeed (p. 36) Hunter Walk recommends that folks avoid \u0026quot;snacking,\u0026quot; [easy and low-impact work,] when they prioritize... [Y]ou're unlikely to learn much from doing them, others are likely equally capable of completing them (and for some of them, it might be a good development opportunity), and there's a tremendous opportunity cost versus doing something higher impact.\nFor data scientist, the snacking is often \u0026quot;pulling data\u0026quot; or other relatively routine work. I do think it's important for data scientists to spend a limited amount of time on this, however, for a few reasons. It's a good way to learn about organizational data and business problems, both of which are critical to be well-versed in. It also builds good will with colleagues in other departments. And it provides an opportunity to say \u0026quot;yes, and\u0026quot; -- to provide the requested data, but also to suggest new higher-impact initiatives that may be possible.\n(p. 39) You should swarm to existential problems, but if a problem isn't existential, then you should be skeptical of adding your efforts where everyone's already focused... Instead, the most effective places to work are those that matter to your company but still have enough room to actually do work. What are priorities that will become critical in the future, where you can do great work ahead of time? Where are areas that are doing ok but could be doing great with your support?\nData scientists are often in the unique position of having insight into what's changing at an organization, because of our closeness to the data. Being able to say \u0026quot;hey, I'm seeing this thing, and here's what I think we should do about it\u0026quot; is a strong place to be.\n(p. 42) Whatever it is, things that simply won't happen if you don't do them are your biggest opportunity to work on something that matters, and it's a category that will get both narrower and deeper the further you get into your career.\nYes, and related:\n(p. 119) [E]arly in your career, you're given well-defined problems, but as you get deeper into it, you'll increasingly encounter poorly defined or undefined problems, and Staff projects will generally start with a poorly scoped but complex and important problem... From that broad, unclear (and potentially wrong) statement, you'll have to identify a concrete approach that works.\nYes, yes, yes. Doing data science at any level involves having a conversation with stakeholders to understand and refine the problem, so that you can tackle it in the most meaningful way. This gets harder or more critical at more senior levels.\n(p. 178) [In an interview with Keavy McMinn from Fastly]\n[Staff-plus engineers] all work on different things, but we have a common goal of taking a holistic, long-term and system-wide view on things. We also try to find and help with the sort of things across engineering that might get overlooked or fall between the cracks. Our CTO supports our work, but doesn't identify the projects to work on, that's up to us.\nAs a Staff-plus data scientist, I really think that it's best to split your time between the vague but important asks that come your way, and initiatives that nobody else is thinking of yet. Of course, we need to build support from management as we go, to ensure that we'll be able to create impact from a project.\nLeadership (p. 75) [Roughly,] management is a specific profession, and leadership is an approach one can demonstrate within any profession...\n[L]eaders have a sufficiently refined view of how things ought to work such that they can rely on their distinction between how things are and how they ought to be to identify proactive, congruent actions to narrow the gap. [Also,] they care enough about the gap to actually attempt those narrowing actions.\nAt the Staff-plus level, data scientists should have a broad enough context to identify and prioritize what needs to change, and a broad enough set of skills (soft and technical) to actually push things forward. This could mean writing a memo to senior management, supported by statistical analyses, or it could mean writing a technical spec to an audience of software engineers.\n(p. 168) [In an interview with Ras Kasa Wililams, Staff Engineer at Mailchimp.]\n[O]nce you become a Staff Engineer, you're a member of \u0026quot;Engineering Leadership\u0026quot;... In my view, it's about thinking globally. That means partnering with other[s]... to understand the company-wide business/product strategy and distill that into an Engineering-wide technical strategy... It's about taking that global thinking and applying it locally.\nEspecially in the \u0026quot;Tech Lead\u0026quot; version of the role, the Staff-plus data scientist is working to keep a team aligned with the company's goals as a whole. This can sometimes cause some disagreements. On the TPT Search team for instance, the three co-leads (myself, the Product Manager, and the Engineering Manager) each represented different goals, and we had to work together to align them. I represented specific pain-points in the user experience and associated metrics; the PM represented current high-level product priorities; and the EM represented software quality and robustness, and the developer experience. We were generally effective at this because we all understood how our work rolled up to long-term success, and because we all communicated effectively and frequently with each other and colleagues on other teams across product development, marketing, and other divisions.\nDefining the Role A thing I've appreciated about becoming more senior is that you get more of an opportunity to define your role, and define what success should look like. Part of that is working with management to write job descriptions, and part of it is helping to define what successful data science at your organization means.\n(p. 171) [Still from Ras Kasa Williams]\n[At Mailchimp, there's] a recurring call for all Staff, Senior Staff, and Principal Engineers meant as a space to surface and discuss problems, assign owners and action items when needed, and generally build community with each other.\nI really like this, and I think there's enough commonality in roles between Staff-plus Software Engineers and Staff-plus Data Scientists that we should join this sort of peer group whenever it exists. Many of the \u0026quot;soft skills\u0026quot; challenges of having leadership expectations without formal management authority are the same. Plus we can talk about the irritating parts of Python.\nThere's much more in the book worth discussing from the point of view of a Staff-plus Data Scientists. If you have thoughts or experiences, I'd love to hear them!\nps -- shout-out to Aaron Schumacher, who excerpted some other great snippets from the book in his blog last year!\n","date":1653609600,"id":"29","link":"/2022/05/staff-data-scientist-comments-on-will-larson-s-staff-engineer-book/","section":"post","summary":"I recently read Will Larson's excellent book Staff Engineer: Leadership beyond the management track. Larson covers the individual contributor (IC, not management) roles that software engineers fill …","tags":["data science","job titles","management"],"title":"Staff Data Scientist: Comments on Will Larson's Staff Engineer Book"},{"body":" Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make the easy cases as simple as clicking a few buttons and looking at pretty graphs. But A/B tests can get statistically complex surprisingly quickly, which is why hiring data scientists with a strong statistics background can make a big difference in the quality of decisions.\nThis is the first in a short series of blog posts about situations that can arise when testing changes that may affect product ratings, an important and ubiquitous part of e-commerce web sites. For certain changes to our site, we may want to understand not just whether users purchase more, but also whether they’re more satisfied with the items they purchase. You care a lot about satisfaction, as it’s closely tied to future purchase rate! There are several ways we could (and should) potentially measure this, such as return-to-site rate, complaint or refund rate, etc. But a direct way to estimate satisfaction is to look at the ratings that users provide.\nHowever, when you’re making changes to an e-commerce site, your interventions can affect ratings in complex ways. This makes A/B testing tricky.\nThe change could increase purchase rate or could change the mix of items purchased. The change could indirectly (via purchase rate or item mix), or even directly, affect the true satisfaction of items acquired. We cannot directly see true satisfaction! The change could directly or indirectly (via satisfaction) affect the rate at which items are rated, either by affecting peoples’ inclinations to rate at any time, or by changing the timing of ratings. The change could directly or indirectly (via the above factors) affect the rating given. Put all of these together, and it’s clear that ratings are a proxy for satisfaction, not a direct measurement of satisfaction, and your decision-making process should treat the data accordingly!\nDelayed and Biased Ratings A challenge with ratings in e-commerce is not just that they’re an incomplete proxy, but they’re also delayed. If you order shoes from a web site, the earliest you’ll rate the item would be when you receive it, and more likely you’d rate it only after several days or weeks. And you might only rate it three months later, when you log back into the web site and are reminded of your past orders and prompted to rate them.\nFurthermore, different people with different experiences differ in how they rate. Higher-priced items, where you spent a bunch of time reading reviews, might lead to a higher likelihood to rate. Items that were defective on arrival, or that were very disappointing, might lead to early, negative ratings.\nWhatever you measure at the end of your A/B test is only an approximation to the eventual ratings that will trickle in, and even those ratings will be only an approximation of the true satisfaction of your customers. All of these complexities affect the validity of the signal you’re measuring.\nLet’s do a quick simulation of the impact of these factors on ratings over time. We don’t need to have a realistic model for now, just a toy model to drive some intuitions. I’ll make this relatively simple:\n10,000 people purchase a pair of shoes. Shoes are either cheap or expensive. Assume the purchase mix is 50/50. People can be unsatisfied, satisfied, or glowing about their purchase. If they rate, they’ll give 1, 2, or 3 stars, respectively. The cheap shoes are 20% likely to be unsatisfactory, and 20% likely to have the purchaser be glowing. The expensive shoes are 10% likely to be unsatisfactory, and 40% likely to have the purchaser be glowing. Assuming the shoes are satisfactory, people will rate the expensive shoes 20% of the time, and the cheap shoes 10% of the time. Assuming the shoes are satisfactory, and if they’ve decided to rate, people will rate early 50% of the time, and late 50% of the time. If the shoes are unsatisfactory or if the purchaser is glowing, the likelihood to rate doubles, and the likelihood to rate early increases to 75%. The early reviews might be what you see after running an experiment, while the later reviews might accumulate months later. There’s no need to simulate an A/B test for now, just the ratings over time. Here’s some R code to simulate this. (Don’t worry if you don’t read R! It just implements the above and makes a graph of three views of the average rating.)\nshoe_options = c(\u0026#39;cheap\u0026#39;, \u0026#39;expensive\u0026#39;) sat_options = 1:3 sim_ratings \u0026lt;- function(ct=10000, cheap_sat_prob=c(.2, .6, .2), exp_sat_prob=c(.1, .5, .4), rate_probs=c(.1, .2), rate_mult=2, late_probs=c(.5, .25)) { ret \u0026lt;- tibble(user=1:ct, shoes=sample(shoe_options, size=ct, replace=TRUE)) ret %\u0026gt;% mutate(satisfaction=ifelse(shoes==\u0026#39;cheap\u0026#39;, sample(sat_options, size=ct, replace=TRUE, prob=cheap_sat_prob), sample(sat_options, size=ct, replace=TRUE, prob=exp_sat_prob))) %\u0026gt;% mutate(rates_prob=ifelse(shoes==\u0026#39;cheap\u0026#39;, rate_probs[[1]], rate_probs[[2]]) * ifelse(satisfaction != 2, rate_mult, 1)) %\u0026gt;% mutate(rates_late_prob=ifelse(satisfaction != 2, late_probs[[1]], late_probs[[2]])) %\u0026gt;% mutate(rates=rbinom(ct, 1, rates_prob), rates_late=rbinom(ct, 1, rates_late_prob)) } sim1 \u0026lt;- sim_ratings() res1 \u0026lt;- with(sim1, tribble( ~ name, ~value, \u0026quot;True\u0026quot;, mean(satisfaction), \u0026quot;Early\u0026quot;, mean(satisfaction[rates \u0026amp; !rates_late]), \u0026quot;Eventual\u0026quot;, mean(satisfaction[as.logical(rates)]) )) theme_set(theme_minimal(base_size=16)) ggplot(res1, aes(name, value, label=sprintf(\u0026#39;%0.2f\u0026#39;, value))) + geom_point(size = 3, color = \u0026quot;black\u0026quot;) + geom_segment(aes(x=name, xend=name, y=0, yend=value)) + geom_text(nudge_y=.15) + scale_y_continuous(\u0026quot;Average\u0026quot;, limits=c(0,3)) + coord_flip() + labs(\u0026quot;Simulation 1: Parameters in Text\u0026quot;, x=\u0026quot;Rating\u0026quot;) Here’s what I get out of these results, and thinking through the toy simulation.\nRatings don’t reflect true satisfaction. In this simulation, only 22% of buyers will rate at all, and only 13% will rate early enough to be observed during an experiment. If there are more people who are glowing than dissatisfied, and if having strong feelings makes you more likely to rate, then ratings will look more positive than the actual level of satisfaction. Early ratings will differ from late ratings. In this scenario early ratings are a bit lower, as they reflect the more negative experiences of people who bought more expensive shoes but were dissatisfied. Implications So, what do we do about this? I don’t think there’s a one-size-fits-all answer. Not all e-commerce business are equal. A company that sells shoes might be lucky to have a 10% rating rate, while subscription services with very loyal customers, such as Rent the Runway or StitchFix, likely have much much higher rating rates. This said, here are some ideas for dealing with timing and low rating rates when ratings are a key metric for A/B tests:\nWhen doing a power analysis, before the experiment starts, be sure to use the number of people expected to rate during the experiment, not the much larger number of people expected to participate in the experiment. Look at historical rating patterns. In your specific case, do ratings accumulated in the first week or two differ dramatically from the eventual ratings, as in this simulation? The more that they do, the less valid metrics such as average rating will be, and the more work you should put in to ensure you’re making the right decisions about the feature. An easy thing to do is simply to wait. If you can, run the experiment longer, and wait longer after the experiment ends, accumulating more ratings, before you make a decision. Look at rating rates. If the intervention made users much more or less likely to rate at all during the window of your experiment, then the average ratings they provide are much less valid. Review your past experiments. If you re-analyze an experiment from 6 months ago, with most of the ratings that will eventually come in, do you get a different answer? Think about mechanisms. You may be tempted to A/B test an intervention where you remind people to rate their purchases a week after they receive their shoes. That test could easily yield a change in average ratings. But it’s important to remember that the change happened after the purchase decision, meaning there’s no way that the intervention caused a change in satisfaction with their purchase! (It still is likely a good idea, though!) There are other issues that arise with A/B testing and ratings. A fun one is: what happens if users purchase more than one item? Are your statistical tests on ratings still valid? Turns out they’re not! In my next post on this topic, I’ll explain and provide a workaround!\n","date":1653177600,"id":"30","link":"/2022/05/a-b-testing-and-product-ratings-1-delays-bias/","section":"post","summary":"Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make …","tags":["data science","analytics","business","statistics","a/b testing"],"title":"A/B Testing and Product Ratings, Part 1: Delays and Bias"},{"body":"Just a quick post here to note a few professional accomplishments:\nI just added a new publication to my vita -- a peer-reviewed conference proceeedings article about abstractions for building repeated, related versions of similar predictive models. Check out some longer thoughts on Medium, or read the full article. Earlier this year, I added an incredibly old project! A paper that I had contributed a bit to in... 2005! finally got published! It has something to do with the way information flows during speech perception... I think... I'll be talking at two Meetups this Fall -- the RecSys NYC Meetup on Sept. 18th, and the Dataiku Data Science Meetup on Sept. 26th. In both Meetups, I'll be talking about (different) aspects of recommendations systems I'm building at WayUp. And I'll be talking at two conferences this Fall too -- the Predictive APIs conference in Boston in October, and DataEngConf in NYC in November. At both of those conferences, I'll be talking about the software architecture aspects of building job recommendation systems that need to provide compelling recommendations just seconds after a user creates a rich profile. ","date":1536019200,"id":"31","link":"/2018/09/new-publications-and-upcoming-talks/","section":"post","summary":"Just a quick post here to note a few professional accomplishments:\nI just added a new publication to my vita -- a peer-reviewed conference proceeedings article about abstractions for building …","tags":["algorithms","conferences","data science","machine learning","meetup","software architecture"],"title":"New Publications and Upcoming Talks"},{"body":"I've been cooking a recipe recently of my own creation that I really like, and there isn't much similar on the internet, so I'm sharing the recipe here. It's a combination of two great things -- hot-smoking fish with wood chips in a stovetop smoker, and the fermented flavors of Hunanese cuisine.\nSmoked fish and duck are common flavors in Chinese cooking. Tea-smoked duck is common on Chinese menus in America, and Chinese cookbooks have recipes like Steamed Smoked Fish with Black Beans and Chiles. But those recipes typically suggest purchasing a smoked trout or similar, and adding additional ingredients. I've had great luck starting with raw fish. There are a few examples of similar techniques, but not that many.\nHere's what I do. I'm not using my technique for rewriting recipes here, as I want to explain some of the details of the process.\nStovetop Smoked Chinese Fish Serves 2-3, with other dishes. Takes 30 minutes.\nThings You'll Need a stovetop smoker and wood chips -- I have the mini Cameron's smoker; oak chips are good for this 1 whole fish of 1 pound or so, scaled and cleaned, or a meaty skin-on filet -- trout is good oil -- scallion oil is highly recommended Shaoxing rice wine Some or all of:\nscallions, in 2\u0026quot; chunks fermented Chinese black beans chopped salt-fermented Hunanese chiles, or dried whole red chiles thinly julienned young ginger Procedure With a sharp knife, cut diagonal slashes through the skin of the fish. If needed for the fish to fit in the smoker, cut off the tail. Douse the fish with shao-xing wine, and let it sit on a plate. In Chinese cooking, this is common for reducing \u0026quot;fishy flavors\u0026quot;. Prep scallions, ginger, etc. Rinse fish and pat dry with paper towels. Lightly cover fish with oil. Lightly salt the inside of the fish (the outside will have plenty of salt from the black beans and chiles). Cover the smoker tray with aluminum foil, then put the grate on the tray. Put the fish on the grate. Put some of the scallion chunks inside the fish. On top of the fish, put more scallions, fermented black beans, and chiles. Drizzle with a bit more oil. (See below for what it looks like.) Put a heaping tablespoon of wood chips in the smoker. Put the tray of fish in the smoker. Put the smoker on a hot flame, with the cover cracked, just until wisps of smoke start to come out. Turn the heat to medium-low, close the cover, and set a timer for 20 minutes (for a whole fish; less for a filet). Serve family style on a platter, with rice and contrasting other dishes. ","date":1526169600,"id":"32","link":"/2018/05/stovetop-smoked-chinese-fish/","section":"post","summary":"I've been cooking a recipe recently of my own creation that I really like, and there isn't much similar on the internet, so I'm sharing the recipe here. It's a combination of two great things -- …","tags":["recipes"],"title":"Stovetop Smoked Chinese Fish"},{"body":" A thing that I do when I cook is to re-write the recipes I’m using (whether they’re from a cookbook or my own invention) onto a piece of paper in a very specific way. I think the approach I use is handy, so I’m describing it here in case you’d like to use it. (Or in case you need more evidence about how weird I am.)\nThere are 4 ideas that I think are important:\nCopying the recipe, by hand, forces me to think through the steps and keeps me from not noticing something important. (“Step 7 – soak a thing for 24 hours…” :sob:) Putting the ingredients in the order I’ll cook them, next to the method and timing, keeps me from having to flip back and forth. Putting preparation steps next to the ingredients, rather than up-front, usually helps me separate preparation (mise en place) from the actual cooking. Being concise is good. Most recipes are wordy, but you only need all the words the first time you read the recipe. These come about from two major inspirations. One is the Cooking for Engineers blog. The author, Michael Chu, uses tables to indicate how to combine ingredients:\nRatatouille from Cooking for Engineers\nIn the above recipe, from this page (where he also has a verbose, photo-heavy alternative approach), you can clearly see the steps, which proceed from left to right, and indicate how to group ingredients or intermediate stages, until you season to taste. This is a very clever approach, and I like its terseness.\nThe other inspiration is the classic Joy of Cooking recipe style. In the photo below from a 1970s edition, the ingredients are interleaved with instructions on how to use them, rather than listed above the steps.\nNo of course I haven’t made the Joy of Cooking recipe for porcupine…\nGiven these two influences, here’s what a recent set of my notes for cooking a Chinese dinner look like:\nSmoked Fish, Sichuanese Green Beans, Sichuanese Cucumber\nI cooked four things: rice (in a rice cooker), a smoked fish dish (which I’ll write about some other time), and two recipes from Fushia Dunlop’s outstanding Sichuanese cookbook, Land of Plenty. The first of them (marked with a 2, as a reminder to cook it second), is the best of example of what I try to do. Each ingredient is listed, with just enough information to prep correctly: 10 oz beans, 2\u0026quot; , or 3 oz gr. pork. Things that are added together are bracketed. Timing or cooking notes are next to the brackets. Once you start applying heat to things, everything happens in order.\nThe cucumber recipe doesn’t quite follow the rules, as I give a prep step for salting the cucumbers and letting them sit as a separate line. And the fish dish doesn’t follow the approach at all, but only because it’s a simple recipe I know well, and I just wanted to remind myself what ingredients I was using. (The rice is in all caps so I don’t forget about it!)\nCompared to the Cooking for Engineers approach, recipes written out this way are more linear, and are sequentially vertically rather than horizontally. Compared with the Joy of Cooking approach, recipes are terser and written around what you do to the ingredients, rather than in paragraphs. And compared to them both, the process of writing before you cook is integral. I don’t think I’d publish a recipe using this style – the extra words of explanation are helpful when you’re trying to communicate with a reader. The act of figuring out how to write a recipe in this way, in the minutes before you start dicing onions, keeps at least me prepared and on-track.\nIf you try something like this, please let me know how it goes! (And if you try any of the recipes above, same!)\n","date":1521936e3,"id":"33","link":"/2018/03/how-i-rewrite-recipes/","section":"post","summary":"A thing that I do when I cook is to re-write the recipes I’m using (whether they’re from a cookbook or my own invention) onto a piece of paper in a very specific way. I think the approach I use is …","tags":["cooking","recipes"],"title":"How I Rewrite Recipes"},{"body":" This is my first new post on harlan.harris.name for a while. The occasion is a change of scenery. For about 10 years, my primary blog has been on WordPress, more recently supplemented by Medium. But WordPress and Medium are limited for technical writing, and the trend among data people recently has been to publish static sites built with Blogdown and Hugo. So that’s what this is.\nThe technology I’m using (more on it below) lets me do fun things like trivially embed math: \\(\\sum_i a^2_i\\), or generate plots with embedded code:\nlibrary(ggplot2) ggplot(cars, aes(speed, dist)) + geom_point() I plan to do more of that over time. If you’re curious, I summarize below the history of my professional web presence, and give some highlights of the way this blog is built, and some notes on migrating my old posts from WordPress and Medium to here.\nHistory of my Professional Web Presence 1995–2009: Hand-written web pages on servers owned by universities I was associated with. 2005–2009: Blog on Blogspot.com. More of a personal blog than a professional one, but I did write about cognitive science research from time to time. 2008–2018: WordPress site, most recently hosted on HostGator. 2009–now: Twitter, which has almost entirely been my professional persona. I post less there than I used to, when I was active with Data Community DC. 2008–now: LinkedIn, which I was never very active on, but I occasionally still share articles. 2015–now: Medium, which has been my primary professional blogging platform recently. I will copy everything I post there to here, but not necessarily vice-versa. New Blogging Stack I write posts in Blogdown, a system for generating static posts. The support in RStudio is pretty handy. I write RMarkdown (with embedded code), then whenever I save, the framework generates normal Markdown. Blogdown works with Hugo, which seems to be a very powerful framework that I’m only scratching the surface of. I’m using a theme called Hugo Bootstrap Premium, with a few minor style tweaks. The code for this blog is all on Github. Please don’t fork my blog and republish it elsewhere. That would be creepy. As recommended by the Blogdown folks, I’m using Netlify to post my blog. It’s pretty easy to use. Whenever I push my blog code to Github, Netlify will see the changes and regenerate the site within a minute or two. To ensure that, when the very sad day comes that Medium.com shuts down, I don’t lose anything, I’m using a tool called mediumexporter to archive my Medium posts to this blog. It works relatively will, but see notes below. Migration Notes Copying my old WordPress posts was a relatively easy manual process. Because Markdown passes HTML through unchanged, I just copied the HTML source that WordPress provided into new posts. Links and embedded images from other sites (such as Flickr) worked without changes. All that was necessary was to write a few lines of header information, and double-check that nothing weird was going on.\nThe publications page was a complex nightmare of nested tags in HTML, so I just copied the text off the web page, and re-created the links when appropriate. The about page is new, and rather needs an “under construction” animated GIF…\nThe mediumexporter tool works relatively well. For each post, I ran the following pattern:\nmediumexporter --headers https://medium.com/@HarlanH/on-how-and-when-to-teach-layers-of-abstraction-in-programming-d220c4b5e5b9 \u0026gt; 2017-10-05-on-how-and-when-to-teach-layers-of-abstraction-in-programming.Rmd I’ve submitted an issue to the mediumexporter project recommending a feature (which I might build myself) to support Blogdown specifically. After running the command line above, much of the remaining, somewhat tedious work was to copy known fields into Blogdown’s header format.\nFortunately, I had some time on an airplane to do the tedious work, and a bit of time on a weekend to finish this post and get everything migrated. From here on out, blogging should be even easier!\n","date":1520726400,"id":"34","link":"/2018/03/on-moving-my-blog-from-wordpress-to-blogdown/","section":"post","summary":"This is my first new post on harlan.harris.name for a while. The occasion is a change of scenery. For about 10 years, my primary blog has been on WordPress, more recently supplemented by Medium. But …","tags":["meta","R"],"title":"On moving my blog from Wordpress to Blogdown"},{"body":" This post was originally published on Medium\nThere’s recently been some interesting opinionated writing in the R statistical programming community about how and when to teach the abstracted, easy-to-use approaches to solving problems, versus the underlying nitty-gritty. David Robinson, Data Scientist at Stack Overflow, wrote a blog post recently called Don’t teach students the hard way first. The primary example was on the data-manipulation tools in the tidyverse, versus the underlying methods in base R, but the discussion was mostly about principles in pedagogy. Some highlight quotes from the original article (which I recommend reading!):\nthat phrase keeps popping up: “I teach them X just to show them how much easier Y is”. It’s a trap…\nfeeding students two possible solutions in a row is nothing like the experience of them comparing solutions for themselves…\nstudents should absolutely learn multiple approaches (there’s usually advantages to each one). But not in this order, from hardest to easiest…\none exception is if the hard solution is something the student would have been tempted to do themselves.\nAnd some comments and responses I found particularly interesting:\nIf there’s an educational purpose for a long and short way, then just be honest from the get-go: “There [is] a shorter way to do this, which I’m going to show you next, but I’m first going to show you the long way since I want you to understand X” — where X is a theoretical point, not nostalgia. — **Joyce Robbins**\nView this post on X (HarlanH) View this post on X (HarlanH) So, teach easier first, unless the harder is more obvious; teach usefulness first; teach the familiar first. All of this makes sense, and I’d like to expand on it a bit, not to disagree, but to try to clarify why I agree so much.\nFrom my 2013 talk where I start with the hardest way to use ggplot2.\nIn R, I certainly agree with David and others in the specific case of the tidyverse , but I’ve previously weighed in on a related issue, taking the other side. In 2010, I wrote a blog post about ggplot2, the extensively-used data visualization library, and the ways that its design was both amazing and confusing. In particular, I criticized author Hadley Wickham’s pedagogical approach in his book on ggplot2, specifically specifically his introduction of the qplot shortcut early on. Subsequently, I’ve given several presentations about ggplot2 where I started at the lowest level of abstraction, the layer, and built up higher levels of abstraction and short-hand over the course of the talk. It’s worth noting that nobody ever uses this lowest level of abstraction in practice, but I thought it worthwhile to show how to construct a graph using that approach, before introducing the shorthand that everybody uses.\nWas I wrong about that? I’m not sure. I want to make a case that both tidyverse-first (in the case of data manipulation) and layers-first (in the case of ggplot2 specifically) are the right answers. To rescue any hope of consistency on my part, I need an answer to the following question:\nWhat is a general principle that would help us determine what to teach first?\nOption 1: Teach the low-level nitty-gritty first, and then build on layers of abstraction. This is inconsistent with my advice — the tidyverse is not the low-level nitty-gritty. Note that in programming more broadly, this might mean teaching assembly language (hardware) and LISP (math) first, then high-level languages later. This approach is a popular way to build technical experts with deep understanding in, say, university departments, but it can be intimidating to many beginners. I don’t think this is the right answer in general, and certainly not for statistical programmers.\nOption 2: Teach the layers you’ll use in practice first, then dive under the hood later. This is also inconsistent with my advice — nobody uses layers directly in ggplot2. In programming more broadly, this would be like teaching working examples in Javascript and Python first, then working backwards to explain why the code works. This is highly appealing to new learners, as you get to see some results quickly. But learning this way can feel like magic, and it often requires a difficult shift in thinking later on. (See also…) \u0026gt; the tidyverse tends to be more consistent than base R — Hadley Wickham\nOption 3: Teach at the layer of abstraction that is most self-consistent and has the clearest metaphors. The tidyverse has excellent, clear verbs for explaining what’s going on with data processing, and is generally self-consistent and complete. Additionally, it maps well to peoples’ understanding of what you can do with data in spreadsheets or other statistical data systems, especially if they’re familiar with SQL. There’s some magic in non-standard evaluation and bare names for things, but it can be hand-waved for a while without too many problems, and it’s only really surprising for experienced programmers.\nSimilarly, the layers approach to writing ggplot2 is self-consistent and complete, and has minimal magic — the inferences that make writing advanced ggplot2 fast and elegant have to be manually written out. But you can (and definitely should) learn the shortcuts and “slangy” abbreviations later.\nDoes this framework — teach the self-consistent metaphors first — make sense? Any readers with a background in learning theory have a critique? I think this is what Dijkstra suggests in the article that Shog9 linked to, at least if the thing to be taught is not “radical novelty”: \u0026gt; It is the most common way of trying to cope with novelty: by means of metaphors and analogies we try to link the new to the old, the novel to the familiar. Under sufficiently slow and gradual change, it works reasonably well; in the case of a sharp discontinuity, however, the method breaks down: though we may glorify it with the name “common sense”, our past experience is no longer relevant, the analogies become too shallow, and the metaphors become more misleading than illuminating. This is the situation that is characteristic for the “radical” novelty.\nThanks to Merav Yuravlivker and David Robinson for comments!\n","date":1507161600,"id":"35","link":"/2017/10/on-how-and-when-to-teach-layers-of-abstraction-in-programming/","section":"post","summary":"This post was originally published on Medium\nThere’s recently been some interesting opinionated writing in the R statistical programming community about how and when to teach the abstracted, …","tags":["programming","R","teaching","computer science"],"title":"On How and When to Teach Layers of Abstraction in Programming"},{"body":" This post was originally published on Medium\nI recently attended two small conferences — the ISBIS (International Society for Business and Industrial Statistics) 2017 conference, held at IBM Research in Westchester County, and the Domino Data Lab Popup, held in West SoHo. I was invited to speak at ISBIS (slides here, if you’re curious), but for this post, I want to summarize some insights from other people’s talks.\nIn chronological (to me) order… First a few talks from ISBIS that I particularly liked (note that I only saw a fraction of all the talks):\nMerlise Cylde from Duke talked about Bayesian Model Average, which is interesting and clearly hard. She used some terminology that was new to me, although apparently it goes back to 2000: M-Closed, M-Complete, and M-Open describe the relationship between the true data-generating process and the space of models you’re exploring. Using Machine Learning theory jargon instead of statistical jargon: M-Closed means that the hypothesis space is known to include the “true” target function; M-Complete means that you know that the target function is too complex, and that your hypothesis space only includes approximations; M-Open means that you have no idea whether your hypothesis space includes the target function or not. I like it.\nJohn Myles White from Facebook talked about A/B testing at web-scale. Best quote: “the thing that is holding us back is computing percentiles.” The problem is that to compute percentiles exactly you normally have to sort the data, which is practically impossible if it’s petabytes long. There are approximation schemes that are fast, but apparently they’re problematic if you’re trying to compute a 99th percentile and you get a number that might be anywhere from the 98th to the 100th percentile.\nZhenming Lin from the College of William and Mary talked about a research paper under review that unifies block model and small-world models of networks. Part of it was over my head, but the intuition was that you could project the high-dimensional set of connections down to a low-dimensional space, then use topological data analysis to recover linear distance, which would either be spiky in the case of a block model, or flat in the case of a small-world model. Seemed clever and potentially useful.\nAnd a few talks from the DDL Popup led to both interesting questions and great insights:\nI’ve been thinking a bit recently about what aspects of mostly-transactional customer service can be automated, and which can’t, and how to do that balance. In a panel discussion about data in the beauty products retail industry, Jenifer Jones-Barbato from Estee Lauder talked about the challenges of translating the “lady at the counter” at a department store to an online recommendation system. What parts of that personal interaction are intrinsically valuable? What parts, when automated, feel creepy?\nAnother thing I’ve been thinking about is how and when data folks can embrace qualitative data. Alex Leeds from Squarespace discussed how they use qualitative data from user interviews to inform what quantitative data to collect at scale. Data science for digital products should be thought of as an extension of UX/Design efforts, I think, with Design leading.\nEduardo Ariño de la Rubia from Domino Data Labs compellingly highlighted all the ways in which we live in an era of Data Science automation. I completely agree, and have in the past argued that the reason why Data Science came about, as a career, around ten years ago, was that the tools got good enough that one person (and in fact, many thousands of people, often with skills learned in academic science) could do the full process from data collection and exploration, modeling, and presentation or integration into a data product, on their own, on a laptop. Previous data mining and predictive analytics processes tended to require larger teams of people with specific skills.\nJon Roberts from dotdash (formerly About.com) gave a talk full of pithy quotes. Here are a few: 1) “Convince people that science can be applied to your problem,” for example by showing that you can answer the question: “is it seasonal or is it real?” 2) “Machine Learning can’t tell you the precision that the business needs.” 3) “Someone talking about your work has to be able to talk about your work.” 4) “Small data matters a lot, lot more than big data.” 5) “Making people laugh with data is the best version of data science.”\nAlfred Lee talked about the role of curiosity in data science, and how it drives hypothesis generation, and thus the creation of knowledge and systems. A great audience question he addressed was about how that trait, curiosity, also leads to a tendency to go down rabbit holes, neglect reproducibility, and neglect data quality issues. He suggested techniques such as paired work and frequent stakeholder (not supervisor) checkins to minimize this tendency.\nMax Shron from Warby Parker talked about how they plan data science projects to set them up for success and impact. They have a ranked list of things that are important, and wisely, they prioritize solutions that fit into a workflow over model accuracy. A useful, used solution beats a perfect, unused solution every time.\n","date":1500163200,"id":"36","link":"/2017/07/a-few-things-i-learned-from-two-small-data-science-conferences/","section":"post","summary":"This post was originally published on Medium\nI recently attended two small conferences — the ISBIS (International Society for Business and Industrial Statistics) 2017 conference, held at IBM Research …","tags":["data science","machine learning","conferences"],"title":"A few things I learned from two small Data Science conferences"},{"body":" This post was originally published on Medium\nOccasionally when chatting with other data scientists, especially with others who are interested in integrating predictive models into production software system, the word “scaling” comes up.\nNot this. Although some West Coast data scientists are into this kind of scaling too.\nI think this is a great question, but it’s a little underspecified. There seem to be at least three qualitatively different notions of “scaling” in data science, and it’s worth the effort to clarify each of them, and address how people tackle them.\nSpecifically, I think the real questions that underlie “scaling” are: “what happens when you have a lot more training data?”, “what happens when you have to make a lot more predictions?”, and “what happens when you have to fit a lot more models?”\nScaling the Training Data How does your system perform if you have massive amounts of training data? Can you fit a model if you have a petabyte of log files? What if the files are on a thousand separate machines — can you learn in parallel?\nThese are some of the classic scaling questions that arose from the rise of web-scale data, and of storing every action from every user on a Hadoop cluster. In some ways, this isn’t a very interesting question. For many practical problems, the best thing to do is to sample the data down to a reasonable size and use an off-the-shelf learner. The additional percentage point of performance may not be worth your time to pursue, even allowing for the adage that more data beats better algorithms.\nThere are important exceptions, of course. If you’re trying to learn to classify images, you probably do need all of Google’s data. Much of the immense progress made in recent years in image and video analysis, and text and speech processing, is indeed due to having vastly more labeled training data than was available 15 years ago.\nWhen you do need all of the data, machine learning algorithms such as Stochastic Gradient Descent parallelizewell. You can also do clever tricks such as split up your learning problem into a slow part and a fast part, and only do the slow part occasionally — Facebook does this with ad click models, using expensive random forest models occasionally to find critical feature combinations, but linear regression nightly to optimize against current data.\nAnd it’s usually worthwhile to parallelize operations such as cross-validation and hyperparameter selection, even when the data is not that large.\nScaling the Feature Space A related subissue is the impact of scaling the number of relevant or irrelevant features (columns, predictors). Penalized models such as LASSO regression have allowed biostatisticians working with DNA sequence data to build statistical models with thousands or millions of potential predictors, but perhaps only hundreds or thousands of rows of data. These algorithms scale well with the feature space, while other algorithms fail to learn at all.\nMy PhD thesis was in this area — I was studying how online linear learners (regression-like algorithms without a memory, learning from a stream of data) perform when the number of irrelevant attributes increases. Standard algorithms required a linear increase in the number of examples as the number of irrelevant attributes grew, but a class of learners that I studied and expanded was able to to learn the same functions with a sub-linear number of examples. Don’t worry if you haven’t heard about this line of research — it’s not that useful in practice!\nScaling the Scoring Process Although it may be a problem if your model takes a week to fit, in practice you might be able to live with that, as long as the model can make predictions quickly. Some models, such as k-Nearest Neighbor or large SVMs, can take a long time to make a prediction, as they have to compare each item to some or all of the training data. On the other hand, classical techniques like regression and decision trees are very fast to score.\nNeural networks, including deep learning networks, can be slow to score, as they have to pass an image or other piece of data through perhaps millions of computations. Fortunately, the same GPUs that have dramatically sped up neural network fitting work well to speed up the prediction step as well, allowing Google and others to offer real-time translation and face detection to the entire world.\nA class of models that presents some scoring scaling challenges are those that underlie recommender systems, such as those that suggest Amazon items for you, or rank LinkedIn’s news feed. As these systems are typically getting new items and users on a continuous basis, collaborative filtering algorithms such as matrix factorization are problematic, as retraining can take a substantial amount of time. The research community has found techniques for keeping collaborative filters up to date efficiently, allowing those systems to be used at scale. YouTube and other systems use a two-stage process, with a quick but imprecise process funneling millions of items down to hundreds, followed by a more expensive process to rank the results.\nIt’s also worth noting that scoring from predictive models is an intrinsically embarassingly parallelizable process. If you need to score 1000 items in a tenth of a second, and it takes a hundredth of a second to score, then just spin up 100 computers, each with a copy of your model.\nScaling the Fitting Process Finally, there is an operational aspect to scaling data science work. Suppose you need to fit many related models, but still need data scientist supervision. We had this constraint at EAB, where each customer got a separately-fit version of a model template, with customizations based on their data and its unique aspects. The scaling problem, then, is not just how long it takes to fit a model, but how long it takes to review the data, to review model metrics, to identify opportunities for improvement and perform model selection, to push the model into production and validate it.\nView this post on X (HarlanH) These sorts of scaling issues are related to Data Scientist time, not CPU time or wall clock time. The use of good, customized support tools can allow a data scientist to perform each step efficiently, using their time effectively to make highly skilled decisions, not perform repetitive or error-prone work. I talked about some of the tools we built for ourselves to do these tasks at the Shiny conference in 2016.\nThe folks at Facebook commented on related issues recently in their announcement of Prophet, their open-source time series forecasting system. \u0026gt; The typical considerations that “scale” implies, computation and storage, aren’t as much of a concern for forecasting. We have found the computational and infrastructure problems of forecasting a large number of time series to be relatively straightforward — typically these fitting procedures parallelize quite easily and forecasts are not difficult to store in relational databases such as MySQL or data warehouses such as Hive. \u0026gt; The problems of scale we have observed in practice involve the complexity introduced by the variety of forecasting problems and building trust in a large number of forecasts once they have been produced. Prophet has been a key piece to improving Facebook’s ability to create a large number of trustworthy forecasts used for decision-making and even in product features.\nSummary When starting a new data science project that will turn into a production system — something with a systems development life cycle, it’s worth your time to think ahead about the scaling challenges you’ll face. If you have huge data, figure out whether you need to use all of it, or if you can sample down to something tractable and still get adequate results. If you will need to make predictions or recommendations in real-time, or in fractions of a second, you may want to choose algorithms that don’t require extensive processing in the scoring phase. And if you’re going to be doing the same process repeatedly, broaden the scope of what you’re considering the problem, and sketch out plans for automation.\nAs data scientists, we’re responsible for building impactful statistical systems. Getting ahead of scaling issues is part of the challenge, and being clear about which aspects of scaling you have to address for a particular system will make your job easier.\nThanks to Brian Eoff for helpful suggestions.\n","date":1496966400,"id":"37","link":"/2017/06/what-do-data-scientists-mean-by-scaling/","section":"post","summary":"This post was originally published on Medium\nOccasionally when chatting with other data scientists, especially with others who are interested in integrating predictive models into production software …","tags":["machine learning","data science","scaling","software architecture"],"title":"What do Data Scientists mean by “Scaling”?"},{"body":" This post was originally published on Medium\nA particularly good way to get a little more out of professional conferences is to blog about your experiences, I think. It makes you focus your thoughts on things like “what’s the big take-away here,” and “what should I be asking people in the hallways?” Rather than just summarizing what you saw, or making snarky Twitter comments (also worth doing!), a great conference blog post is synthesis — combining insights from multiple presentations and conversations into a coherent new whole that helps clarify ideas.\nI recently returned from the INFORMS Analytics 2017 conference. INFORMS is the professional society of Operations Research — if the field is not familiar to you, think about a 1950s-style combination of industrial engineering, business, and theoretical computer science, focused on efficiency in big industrial and government problems. They have a big academic conference, but the Analytics conference is focused on business and practice, not theory. I like a bunch of aspects of OR’s culture, as a neighbor to data science, so I’ve attended a few times. In 2014 and this year, I was an “official” conference blogger, meaning that my posts were on the official conference web site, and my name badge had a fancy “Blogger” ribbon on it. I’m not sure if I managed to synthesize any big new ideas this year, but it was a fun, valuable, and recommended experience.\nHere are links and pull quotes from the four posts I wrote. Click through to read them in their entirely, if you’d like!\nOPERATIONS RESEARCH, FROM THE POINT OF VIEW OF DATA SCIENCE (REDUX) …the value of OR’s participation in the broader analytics community is, I would argue, to bring decades of maturity to the table, and help focus people and organizations on the bottom line — results that matter.\nEXPECTED UTILITY AND PRIVATE ISLANDS: HOW TO GAMBLE The Megabucks machine has odds of 1 in 49,836,032, and you could win $10 million. Clearly, the expected value is garbage, but still, you could win enough to ditch this conference and buy a (small) island. So that’s what I recommend you do. If you do win, you’re taking me to dinner at Nobu.\nGUEST POST: PROBLEM DEFINITION AND WORKFLOWS IN ANALYTICS SUCCESS This is a guest post by Michael Zargham, who gave a sadly under-attended but content-rich talk this morning. I thought his framing was spot on around how to set up an analytics group to repeatedly and efficiently explore potential new “well-defined, solvable, impactful” problems, determine what should be built, do the research to actually get the problem solved, and deliver it in the best possible way to get the desired results.\n“WHAT SHOULD I DO,” REALLY… Here’s the situation you don’t want to be in: I tell you to bring an umbrella a couple of days in a row. But it didn’t rain. You start to question your trust in me. You say, “hey, why’d you tell me to bring an umbrella?” I shrug. You lose your trust in me and stop listening to my guidance.\nAnd finally, since the conference was in Las Vegas, here’s a photo of my new socks:\n","date":1491696e3,"id":"38","link":"/2017/04/conference-blogging-informs-analytics-2017/","section":"post","summary":"This post was originally published on Medium\nA particularly good way to get a little more out of professional conferences is to blog about your experiences, I think. It makes you focus your thoughts …","tags":["data science","conferences","analytics","operations research"],"title":"Conference Blogging INFORMS Analytics 2017"},{"body":" This post was originally published on Medium\nA particularly good talk at Strata NY last year was by Brett Goldstein, former CIO of Chicago, who talked about accountability and transparency in predictive models that affect people’s lives. This struck a strong chord with me, so I wanted to take some time to write down some thoughts. (And a rather longer time to publish those thoughts…) I’m sure others’ have thought about this more and have better takes on this — please comment and provide links!\nA slide from Goldstein’s Strata presentation.\nThere has been a lot of discussion recently about accountability in predictive models, and the failures of certain systems to avoid troublesome racial bias, in particular. ProPublica wrote an insightful report about a criminal justice model that tries to estimate the likelihood of recidivism, to help courts with sentencing, but that appears to treat racial minorities more harshly. Cathy O’Neil has a new book out about all the ways that unaccountable algorithms can hurt rather than help society. Computer Scientists are trying to find ways of measuring and correcting disparate impact in predictive algorithms. For those of us who build models that affect peoples’ lives, we need to be highly attuned to this issue, and focused on finding ways to do the right thing.\nIn general, transparent algorithms have the advantage of, potentially, more eyes on them, reducing the likelihood that an error or oversight results in a bad outcome. Complex algorithms — and machine learning systems are often quite complex — can be fragile in ways that lead to subtle failures and poor predictions. Because of this, Goldstein asserted that there will be a push by customers of these algorithms (governments and public institutions, mostly) to require that the predictive models be open to inspection and validation.\nThese are top-down reasons for transparency in predictive algorithms — the people who are writing the checks are wanting to know that the system works as it should. But there are also bottom-up reasons for transparency, and that’s the development of trust. In many cases, predictive models do not directly translate into a decision or policy, but instead inform a human who must take the prediction into account while making a decision. People using these predictions want to trust that the models are working well, and will help them do their jobs better.\nTrust can arise from several avenues. Providing the code or algorithm is not likely to be helpful for most non-technical end-users, but in some limited cases that it could be. In more cases, access to the people who implemented the system, and the ability to constructively ask questions, can be highly reassuring, although this approach clearly doesn’t scale. Also, just as many people trust open-source software simply because of its openness, transparent predictive models might be more trustworthy simply because the openness helps keep the data scientist honest.\nWhere does this put data scientists and companies who want to create businesses based on predictive models? It seems to me that a fully open system can only rarely be the basis of a business with recurring revenue. (A consulting business is profitable, but does not have recurring revenue.) That is, if you’re creating a predictive model of, say, hospital readmission likelihood, you’d like to be able to sell predictions of readmission, for individual patients, on an ongoing basis. If instead you were required to sell and share an interpretable model — say one of MIT Professor Cynthia Rubin’s 3x5 card-sized rule lists — you would have to charge up front for the model, which is not likely to provide enough revenue to keep your business going. Trying to sell the same customer a slightly better rule list the next year is not too likely to be successful. (Note that if the predictions are part of a larger, proprietary system such as an otherwise-compelling workflow tool, you may have a shot.)\nAn example of a very interpretable, not very predictive, and not at all proprietary predictive model. (source)\nSo, there’s an inherent tension here. You want to sell predictions, not the model, to institutions that want to know what you built, so they can ensure it’s making predictions that are ethical and aligned with their goals. And you want the people using your model to make decisions to trust the model, so they can get the full value out of it. But you have to maintain some sort of competitive advantage, so your customer can’t trivially take your model in-house, and your competitors can’t just copy the model off of a web site. How do you resolve this?\nTo ground this, let me briefly review what several previous employers of mine do, focusing on three aspects of transparency: the data used, the way the data is transformed (feature engineered), and the core model itself.\nA few years ago, I worked for Sentrana, a company that provides sales and marketing software-as-a-service to large enterprises, mostly in the food manufacturing and delivery vertical. The outputs of the model were mostly recommendations: of prices, or cross-sell opportunities, or similar. For Sentrana, the data was primarily the customer’s own data, supplemented by some open (i.e., government) and proprietary (3rd party) data; there was extensive proprietary feature engineering to get the data into shape; and the models themselves were mostly highly proprietary, based on intense analysis of the underlying microeconomic processes. The results were impressive, but not at all transparent to the customers. As the software was sold to the upper management of very hierarchical businesses, the customer was in a position to rely on high-level improvements in business results, rather than model transparency, and to insist that lower-level employees use the recommendations in their day-to-day work, even without transparency providing trust. Because of the proprietary nature of the feature engineering and models, the customers could not take the model in-house easily, and there was substantial competitive advantage.\nUntil recently, I worked at EAB, a company that (among other things) provides a workflow tool for college academic advisors that includes a predictive model of student success (graduation). The output of that model is primarily a likelihood of graduation, which can be used to identify and intervene with students at risk who may fall through traditional GPA-threshold cracks. The data is mostly the member institutions’ academic data, and the company states openly to its customers (when technically sophisticated people ask) that the core model is more-or-less an off-the-shelf penalized logistic regression with standard best-practices for training, but the feature engineering and framework required to use that off-the-shelf model to get good results is proprietary and necessarily quite intricate. Much of the feature engineering and the specific way the core model was applied remains a trade secret and a barrier for both member institutions and competitors to replicate it. A potential trust issue is that the actual predictions of the model (“this student is at high risk of not graduating”) are not transparent or explainable to end-users. A great deal of effort has been expended by the firm to convince advisors to trust the models, and to use the results to prioritize their efforts with students. From the point of view of the EAB Data Science team, we could have tried to use maximally simple, interpretable models, but we preferred to focus on improving model predictive quality and flexibility, maintaining the value of the proprietary components of the system. That’s probably the right call, but it’s not an easy one.\nI recently worked for a while at an HR analytics software-as-a-service firm that provides predictions of tenure for job applicants, to reduce the cost and impact on service quality of turnover. In that case, the data is primarily proprietary — the results of a screening survey taken by job applicants, supplemented by some of the client’s data. The feature engineering is also proprietary, so even though the clients know what questions are asked, they don’t know how to make use of those questions. As with EAB’s model, the value of the system and the barrier to entry is the way the company uses its own data, and transforms that data. The scores generated by the model are used by the hiring teams as an informative data point in their hire/no-hire decisions, improving outcomes, but as of when I worked there, did not come with an explanation.\nIt seems to me that you can create a business with recurring revenue (i.e., not a consulting engagement) as long as some aspect of the data → feature engineering → model pipeline is proprietary and hard to copy. A fully proprietary model has little hope of transparency, and will probably be buggy to boot. A great situation is if you have proprietary data. Google is more than happy to tell you all about how they build fancy deep learning systems for photo and speech recognition, translation, and many other things. You don’t have their data set, though, so you can’t possibly replicate their results, even if you have their tools.\nBut in many business cases, as a predictive model vendor, you’re using the customer’s own data. If you’d prefer to use an off-the-shelf model — and you probably should — then your only option for preventing your customer from reimplementing your algorithm is to carefully and insightfully apply your “substantive expertise” to the problem, building engineered features better than anyone else can, and to preserve those insights as trade secrets.\nI’d like to finally mention a recent new technique that might allow end-users to have partial transparency, and to develop trust, even while proprietary aspects of the algorithm remain secret. LIME (Local Interpretable Model-Agnostic Explanations) was developed by Marco Tulio Ribeiro at the University of Washington, along with several colleagues. Briefly, it identifies, on a case-by-case basis, what few aspects of the input, when changed, would most substantially yield a different output. When applied to a prediction from a student graduation model, for instance, it might identify a GPA trend and a major as being the most relevant pieces of information, for a specific student. This is in contrast to traditional variable importance measures, which identify the most relevant inputs overall, which may not provide much insight into a particular case. LIME promises to provide some insight into individual predictions, which may help provide trust (or reasons to ignore a particular outlying prediction, which is also valuable) for end-users. Importantly, the use of LIME doesn’t allow the model to be easily reverse-engineered, and it should be possible to provide explanations at the level of the raw features, not the engineered features. For instance, staying with the student graduation model, if the particular way that GPA trends are calculated and used is proprietary, then LIME may be able to say that a student is at high risk because of the 3.1, 2.8, and 2.5 GPAs of the last 3 terms, without exposing the actual engineered input to the model.\nI haven’t yet had a chance to apply LIME to any systems I’ve worked on, but I’m interested to try it out. If any readers have experience with LIME or related techniques in production in a software-as-a-service setting, I’d love to read your thoughts.\nThanks to Carl Anderson and Brett Goldstein for very helpful comments on a draft of this post!\n","date":1488153600,"id":"39","link":"/2017/02/transparency-trust-and-proprietary-predictive-analytics/","section":"post","summary":"This post was originally published on Medium\nA particularly good talk at Strata NY last year was by Brett Goldstein, former CIO of Chicago, who talked about accountability and transparency in …","tags":["data science","machine learning","algorithms","ethics"],"title":"Transparency, Trust, and Proprietary Predictive Analytics"},{"body":" I, Harlan D. Harris, hereby commit to the neveragain.tech pledge. Please stand with me and hold me to it.\nIt starts: We, the undersigned, are employees of tech organizations and companies based in the United States. We are engineers, designers, business executives, and others whose jobs include managing or processing data about people. We are choosing to stand in solidarity with Muslim Americans, immigrants, and all people whose lives and livelihoods are threatened by the incoming administration’s proposed data collection policies. We refuse to build a database of people based on their Constitutionally-protected religious beliefs. We refuse to facilitate mass deportations of people the government believes to be undesirable. (read the rest)\nPer the neveragain.tech web site, they no longer add signatures to the site, but instead ask people to post a commitment on a personal blog or Twitter, which I have done here.\n","date":1483833600,"id":"40","link":"/2017/01/neveragain-tech/","section":"post","summary":"I, Harlan D. Harris, hereby commit to the neveragain.tech pledge. Please stand with me and hold me to it.\nIt starts: We, the undersigned, are employees of tech organizations and companies based in the …","tags":["data","ethics"],"title":"neveragain.tech"},{"body":"About This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.\n","date":1480945260,"id":"41","link":"/about/","section":"","summary":"About This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.\n","tags":[],"title":""},{"body":" This post was originally published on Medium\nWhen building a complex system, it’s often helpful to think about the design of that system using patterns and abstractions. Architects and software engineers do so frequently, and the experience of implementing predictive modeling pipelines has recently led to a variety of patterns and best practices. For instance, when dealing with large amounts of streaming data, some organizations use the Lambda Architecture to handle both real-time and computationally-intensive use-cases.\nI recently attended the Strata conference here in NYC, where one of the better presentations was by Jon Morra, at the time at eHarmony — A Generalized Framework for Personalization. He discussed a number of interesting topics, but I’d like to focus on his use of a representation-focused abstraction for the components of a predictive model.\nOne of Jon’s slides looks like this: Model[A, B] =: f(A) =\u0026gt; B. This bit of formalism defines a predictive model as a function with two type parameters, A and B, where A is the type, or representation, of the input, and B is the output of the model. This is sort of obvious, but Jon then puts this into a broader context, which is where the insight arises.\nfrom Jon Morra\nFor a predictive model to be well-integrated with consuming applications and other supporting tools, it’s critical that the representations that are used to communicate between systems are well-defined and consistent. Specifically, the application will represent an entity to be scored, provide that to the predictive model, and expected back a score. In a Service Oriented Architecture, this is the Standard Service Contract principle — define a data schema to be used consistently between services. Critically, the entity to be scored is represented using the structures and semantics of the application domain, not the model’s internal domain. I’ll refer to the type, or representation, of this entity as DomainEntity.\nBeyond this relatively basic insight is the idea that there are two other important types. To explain in more detail, I’ve expanded and clarified Jon’s diagram using CamelCase variable names instead of letters. (As a programmer, single-letter variable names are anathema! I don’t know why mathematicians insist on using them so extensively.)\nThere are two processes, with shared representations and some shared processes. First (black) is the Training process. Each (green) step in Training is a functional — a system that takes a set of data and a set of parameters, and generates a new function (blue). Together, the generated functions define a pipeline for Scoring, and each Training step has an analogous step in the Scoring process.\nThe first step is Feature Transformation — the process of converting from a DomainEntity representation to a ScorableEntity representation. Critically, this step happens inside the model — a change to how features are generated or how the core learner works should (must!) require no change to how the application provides an entity to the system. Your DomainEntity could be text, or a hierarchical structure, or something else, but your ScorableEntity is likely some sort of vector of numbers.\nWorth noting is the insight that, according to this abstraction, the DomainEntity objects that are used for training and scoring must be the same. To implement this in practice means either having two pieces of code (one in training, one in the application) that do essentially the same thing, or that there should be a shared service that provides DomainEntity objects (either historical or current) upon request.\nThe second step is Prediction. The process of executing the Predictor Generator to create a Predictor is usually called “learning.” The input to learning is a set of parameters and a set of ScorableEntities, and the output is a Predictor that can generate a RawPrediction, given a ScorableEntity. This is the core of the model, although architecturally it’s just a subset of the overall system. Ideally, the Predictor Generator is off-the-shelf or a minimally-wrapped version of a standard learning algorithm.\nThe third step is Finalizing — a transformation from the RawPrediction to some sort of DomainScore. Frequently, the output of the Predictor is not appropriate for display to an end-user, but must be transformed in some way. This could be normalization, or ranking, or combination with the outputs of other systems. For example, at a previous job, the (basic) RawPrediction we were generating was a likelihood that a college student would graduate. But this underlying probability score was binned into institution-specific red/yellow/green categories for display, based on how that specific institution wanted to think about and talk about risk.\nNote in the diagram that the Finalizer has access to the raw DomainEntity properties, but not to the ScorableEntity features. The Finalizer is implementing business logic, not predictive modeling, so it should process data at the domain level. The abstraction suggests that Finalizing should be thought of, and architected as, a separate module, or perhaps even a separate microservice.\nIt may be that thinking about a predictive system in these terms could help you make better decisions around system architecture and representations. Of course, abstractions are just that — a way of organizing thinking. This may not be the best-possible abstraction. And even if it was, real-world concerns would mean you wouldn’t necessarily implement an abstraction exactly as written. And you shouldn’t re-architect a system just for the sake of elegance, if it’s working and not causing problems. Still, being aware of what good patterns look like and what a good direction to move in is critically important when evolving and growing a complex system over time.\n","date":1478476800,"id":"42","link":"/2016/11/insights-from-a-predictive-model-pipeline-abstraction/","section":"post","summary":"This post was originally published on Medium\nWhen building a complex system, it’s often helpful to think about the design of that system using patterns and abstractions. Architects and software …","tags":["data science","machine learning","software engineering","software architecture","programming"],"title":"Insights from a Predictive Model Pipeline Abstraction"},{"body":" This post was originally published on Medium\nYou’re a data scientist, and you’ve got a predictive model — great work! Now what? In many cases, you need to hook it up to some sort of large, complex software product so that users can get access to the predictions. Think of LinkedIn’s People You May Know, which mines your professional graph for unconnected connections, or Hopper’s flight price predictions. Those started out as prototypes on someone’s laptop, and are now running at scale, with many millions of users.\nEven if you’re building an internal tool to make a business run better, if you didn’t build the whole app, you’ve got to get the scoring/prediction (as distinct from the fitting/estimation) part of the model connected to a system someone else wrote. In this blog post, I’m going to summarize two methods for doing this that I think are particularly good practices — database mediation and web services.\nThere are a few reasons why this matters. First of all, it’s highly unlikely that the world you care about is static. You’re going to have to refit and re-integrate your models, likely pretty frequently. If iteration is painful, it’s not going to happen, and the model’s value will stagnate. Fast, easy iteration is critical. Secondly, the system you’re integrating with probably isn’t static either. If it changes, will your model stop working? (Or worse, start predicting garbage?) A resilient architecture is also critical.\nOverall, you need what software architects call “separation of concerns.” You don’t need to know the font your prediction is rendered in, or even if it’s text or a graphical widget. And the app needs to know how to request a credit score/flight price/list of people you may know. But it doesn’t need to know how you’re computing that. Most importantly, it really (really) doesn’t need to know how you’re doing feature engineering. The interface must happen before you transform the raw entity data into whatever format goes into the mode.\nOn a deeper level, Conway’s Law, an insight from the 1960s, is highly relevant: \u0026gt; Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.\nThis means that if your application Software Engineers are separate from your Data Scientists, and they likely are/probably should be, then your software will have an interface that reflects the handoff between those two teams. So you owe it to yourself and your colleagues to design that interface in a way that enables fast iteration, resiliency, and a good separation of concerns.\nOption A — use a database. If the entities (people, flights, whatever) you need predict don’t change very often, and in particular, they don’t change or suddenly appear right before you need to make a prediction, then I recommend you consider a good old-fashioned database. The application is probably storing everything in a nice, clean, normalized set of tables. In many cases, using those tables is the simplest and best option.\nA good approach that I’ve used in production is to use an equally-venerable technology: batch jobs. Every hour or every day, have a script run that pulls entities from the application database — maybe just the ones that have changed recently — transforms them into the format needed for prediction, runs the prediction method, then writes the results back to the database.\nThe advantages of this are substantial. You can update your model, or even totally replace it, as long as it writes to the same prediction tables the same way. No need to coordinate with the application’s release cycle in most cases. Tools for reading and writing databases are mature in any language you’re likely to be writing a model in.\nA database-mediated architecture diagram I designed and implemented a while back.\nThere are complexities, of course. You need monitoring. If some input causes your model to crash before it writes predictions, you don’t want to learn about it weeks later. So use good error-logging practices and set up a log-monitor that will alert people in case of a problem. You should be evaluating the results, both from a statistical point of view (an accuracy measure) as well as a utilization point of view (are people using the feature and do they come back?). As Ruslan Belkin from LinkedIn says: \u0026gt; There’s only one right answer and starting point for a data product: Understanding how will you evaluate performance, and building evaluation tools.\nAnother complexity is that if you’re reading from the application database, you’ve potentially created a problem where the application developers’ upgrading the database schema can break your model. Be very careful, use good database schema migration practices and testing, and consider creating a separate database schema, owned by the data science team but populated by the application, just for model fitting and scoring.\nOption B — web services. Great, you say, but my model needs to make a prediction for a person who just connected their Facebook account 30 seconds ago, so a batch job isn’t going to cut it. In that case, use the standard pattern of complex software systems these days, a web service. Leveraging some of the technology of the web (HTTP), along with some standard patterns (REST), service-oriented architectures/microservicesassume that each development team is creating a standalone subsystem that talks to other subsystems over a network connection.\nWhat’s needed to make this happen? First, the application can no longer be passive. It has to create a description of an entity to be predicted that your model can consume. Most likely, this description should be JSON or XML, and the syntax and semantics must be clearly documented. How do you represent time? Should the application send a null if information is missing, or a 0? You know the answer is a null, but those programmers will probably send you a 0 if you don’t clarify.\nSecond, your application should apply the same feature engineering during real-time scoring as it did during model fitting. This is a tricky problem. When creating the model, you probably loaded data from relational databases or Hadoop and created something like a data frame or matrix. But if you’re scoring with a web service, you don’t have a database, just a JSON object, so your feature engineering logic won’t work.\nThe student success (likelihood of graduation) model that my team and I at EAB built avoids this problem by converting database tables to a JSON-equivalent object during model fitting, even though it’s not technically necessary. But by doing so, we can use literally the same code to build the model matrix used for scoring as we did for fitting, which reduces the likelihood of hard-to-detect scoring bugs.\nArchitecture of EAB’s Student Success model, roughly.\nThird, your application is now part of a real-time user experience. Congratulations! Now you get to think about DevOps, reliability, and response times, as if you were a computer scientist! There are a couple of options here. If you’re writing in Python, it’s not that hard to write a simple Flask web app that accepts a JSON object, predicts with it, and returns the result. Of course, the real world can make that an inadequate solution. What if you have lots of services, not just one? What if your server needs to go down for maintenance, or gets overloaded?\nThere are several commercial tools available to make this easier. I’ve had good experiences with YHat’s ScienceOps, which supports a cluster of individual R or Python models wrapped in Docker containers for safety. Domino Data Lab has a similar system, as does Microsoft(via their purchase of Revolution).\nSo, I’ve presented two good patterns, neither of which are what you might call simple. If these are good patterns, what are less-good patterns?\nA common one that doesn’t give you the ability to iterate rapidly is to throw a model over the wall, and ask Software Engineers to re-implement it in C# or Ruby or even in SQL. This might be possible for a simple linear model, but if you find later you can get much better results with a Deep Learning model, now you’re going to have to get the development team to refactor their entire architecture, which is going to take months, best case. Don’t throw models over the wall — it violates Conway’s Law and will definitely hurt you long-term.\nA seemingly promising approach is to extract the coefficients or structure of the model into a standard format and provide them to a system that knows how to predict using that format. The PMML (Predictive Model Markup Language, an XML standard) attempts to do this. But it has a generally-fatal flaw in that it doesn’t support much feature engineering. If your entities have any sort of hierarchical or temporal or spatial structure, you’ll have to put the feature engineering on the wrong side of the interface, beforeU the PMML scorer, inside the core application. And now if you ever need to change anything, you have to get in line behind everyone else with a feature request. Bad news for you. There’s a new descendant of PMML that attempts to address this issue — PFA (Portable Format for Analytics) supports pre-processing and post-processing. It will be interesting to see if it takes off.\nAnother anti-pattern is to embed a model written in one language inside an application written in another, using a tool like RinRuby to pass data back and forth. Seems like a good idea at the time, but now your deployment environment has to support two environments, with all the dependencies and libraries that may be required. Testing for new deployments becomes a lot harder, too, and you’re tied to the application developer’s deployment cycle, which will make you much less flexible.\nOverall, I think you’re best off trying to design a database-mediated or web-service based approach. There may be exceptions and good alternatives, especially if you’re in a Big Data environment, or if your data product needs to be rendered as a complex graph. But for the traditional medium-data predictive model, with a substantial systems development lifecycle, focusing on making the application/model interface as clear as possible is your best bet.\nLike what you read? Find the intersection between software engineering and data science compelling? EAB is hiring! As of when this article was posted, we have open roles in Engineering-focused (type B) data scientists, at both more junior and more senior levels. Help us build systems used by students and staff at about 200 colleges and universities!\n","date":1468281600,"id":"43","link":"/2016/07/patterns-for-connecting-predictive-models-to-software-products/","section":"post","summary":"This post was originally published on Medium\nYou’re a data scientist, and you’ve got a predictive model — great work! Now what? In many cases, you need to hook it up to some sort of large, complex …","tags":["machine learning","data science","software engineering","R"],"title":"Patterns for Connecting Predictive Models to Software Products"},{"body":" This post was originally published on Medium\nYesterday was the 2016 National Day of Civic Hacking, a Code for America event that encourages people with technology and related skills to explore projects related to civil society and government. My friend Josh Tauberer wrote a thoughtful post earlier about the event called Why We Hack —on what the value of this sort of event might be — please read it.\nFor my part, this year I worked on one of the projects he discusses, understanding the impact of DC’s rent stabilization laws and what potential policy changes might yield. As Josh noted, we discovered that it’s a hard problem. Much of the most relevant data (such as the list of properties under rent stabilization and their current and historical rents) are not available, and have to be estimated. Getting to a realistic understanding of the impact of law and policy on rents seems incredibly valuable, but hard.\nSo I spun off the main group, and worked on an easier but much less ambitious project that could potentially be useful in just an afternoon’s work. Instead of trying to understand the law’s effect on actual DC rents, I built a little tool to understand the law’s effect on a rather unrealistic set of simulated apartment buildings. Importantly, I did this fully aware that I’m not building with, I’m tinkering; my goal was to do something fun and interesting that might lead to something substantial and usable later, probably by someone else.\nScreen cap of my hackathon app.\nAlso importantly, the statistician George Box famously said that “all models are wrong, but some are useful.” My hope was that, for people interested in the effects of policy on stabilized rents, being able to explore the impact on a small, toy domain might allow them to understand the complex interactions of the law and the world more cleanly. Scientists do this sort of thing all the time — they use models to clarify issues, even when they don’t intent to fully simulate the complexity of the real world.\nSo what does the little app do? It simulates the effects of three key set-in-law numbers on a fixed set of apartments over time. To make things easy, the apartments are in three different sizes of building (5-unit, 25-unit, and 200-unit), each unit is identical, and each unit starts identically at $1000/month. (In DC? Hah!) The important numbers are (1) the baseline inflation rate and annual rent increase rate, usually a few percent, (2) the usual amount rent can go up on turnover, currently 10%, and (3) the maximum rent can go up on turnover to keep units within the same building at similar rents, currently 30%.\nThe second and third numbers are tricky and important. Consider a building where, if one unit happens to turn over frequently, its rent could go up 10% every year or two, compared to a unit with a longstanding tenant increasing only a little. After 5 or 10 years, the two identical units could have dramatically different rents. The law allows the rent on the second unit, when it turns over, to be increased more than 10%, to match the similar unit. This increase is capped, however, at 30%.\nThe rents the city ends up with over time depend on these numbers, as well as on the frequency with which tenants leave and units turn over, and the mix of small/medium/large housing stock.\nJust thinking about the interactions among all of these issues is hard enough. Thinking about them in a more realistic situation, with judicial exceptions, changes in the law and the set of stabilized buildings, second-order changes in the housing market and both renters’ and property owners’ behavior, variations among neighborhoods, and variations in initial and historical rent, although fascinating, seems implausibly difficult to include, at least for now.\nThe model itself is a microsimulationmodel. I literally pretend that each unit exists, and simulate the process of tenants leaving and rents changing over a period of years. (Computers are fast — the simulation takes just a fraction of a second.) I implemented it in a framework that makes it easy to deploy to the web and to interactive change the parameters and assumptions of the model, including the three critical policy knobs mentioned above, as well as assumptions about the world that strongly affect the outcome, such as housing stock mix and the distribution of tenant turnover rate.\nTry it out. There are three buttons near the upper-right that let you simulate policies of a lower bump due to vacancies and a lower cap on big changes to normalize rents within a building. Both policies have similar effects, modestly lowering rents, but substantially decreasing the variance of rents.\nThere’s much more to do to make this actually useful for policy experts or policy makers (pull requests happily accepted!), but even as-is, I think it nicely illustrates the value that formal models make in our ability to understand complex problems.\nIn this case, it seems like if the goal is to reduce the uncertainty and variance among similar rents, then two useful directions might be to reduce the 10% and 30% bumps on vacancy, while slightly increasing the baseline rent increase. Of course, this will prioritize interests of new renters over existing ones, which might or might not be what you want to do. Other policies might differentially affect renters in smaller or larger buildings, or might incentivize better or worse behavior by landlords. Tools like this can help policy-makers develop intuitions and make good decisions.\nThanks to Josh Tauberer for comments on a draft of this post, and for organizing this topic and helpful resources; and Justin Grimes for helpful UX feedback as well as co-organizing C4DC and the NDoCH event.\n","date":1465084800,"id":"44","link":"/2016/06/simulating-rent-stabilization-policy-at-the-national-day-of-civic-hacking/","section":"post","summary":"This post was originally published on Medium\nYesterday was the 2016 National Day of Civic Hacking, a Code for America event that encourages people with technology and related skills to explore …","tags":["R","Shiny","housing","civic hacking","open data","hackathon","Washington DC"],"title":"Simulating Rent Stabilization Policy at the National Day of Civic Hacking"},{"body":" This is an updated version of an article first published on Medium on Oct. 24, 2015.\nI love my smartwatch, way more than I thought I would when I bought it, over a year ago. It’s a Moto 360, which is still better looking than the Apple watch, I think. Why do I love it? It’s not the health monitoring. I turned that junk off as soon as I got the thing. Do not care. It’s because it separates my phone and its alerts (and temptations) from my interactions with other people. The killer feature for smartphones is the vibration notifications. Phone in my bag or across the room? No problem, I can feel a phone call, or a text message, or a news alert, without anybody else knowing. I even downloaded an app that taps my wrist at the top of every hour, giving me the same sense of time as the dorky digital watch I had in 6th grade, but without annoying the people around me. The alerts are mostly different — apps can choose their duration and pattern — and with practice you can tell a few of them apart. It’s fantastic, and even better is the fact that I can see, dismiss, and briefly respond to alerts if necessary, by looking at my watch and interacting with it. Or, better yet, I can choose not to, but still know the type of alert to expect next time I’m not engaged in something more important. But one key function isn’t perfect, and it highlights a limitation of having vibration notifications on your wrist. The broken UX is turn-by-turn navigation. The navigation app buzzes your wrist whenever you have to turn. But then the next action you have to perform is to look at your wrist. Maybe tolerable when walking, but inadvisable when driving, and particularly dangerous when biking. (Update: The Apple Watch gives you different patterns when you have to turn left versus right, which would be a useful, if limited, enhancement to Android Wear.) What if you could feel what direction you have to turn next? What if you could just know when and when to turn, without having to listen to your phone, or learn complex tap patterns on your wrist? Android Wear should support a secondary bracelet for your other wrist. No screen, just bluetooth and a buzzer. Now, you get twice as much bandwidth when apps want to communicate with you. Apps can buzz both wrists at once, or one then the other, or any other pattern. And even better, spatial apps such as navigation can guide you in the right direction, right from the start. Left buzz? Turn left now. Right buzz? Turn right. Both together? Maybe you’ve arrived! What would it take to make this happen? Well, Google would have to make changes, I suspect. The current Vibrator API for Android Wear appears to make the strong assumption that there’s a single vibrating device. Android Wear would have to specifically support multiple, coordinated worn devices with independent vibration support, and probably would have to make additional changes to support Wear devices without a screen. Once Wear supported these devices, though, they’d presumably be easy to manufacture, and we’d see metal smartbracelets, hipster smartbracelets made out of braided leather, and who knows, maybe smartanklets too! Speaking of anklets, a predecessor to this idea is the vibrating ankle compass, whose wearers always know which way is North. Apparently it was transformative to wearers. (Update: There’s been some efforts to sell a Smart Shoe along these lines.) What other devices could increase your communications bandwidth? Google Glass was a failure, and having bluetooth things talk in your ear is now entirely passe. People don’t want audio or visual connections to the internet all the time, it turns out. (And even more, people don’t want you to have audio or visual connections to the internet all the time!) But the very-low-bandwidth notifications from vibrating devices may give people just enough connectivity to know what they need to know, without interrupting their interactions with the real world. Google? (And Apple, I suppose…) Get started! ","date":1464566400,"id":"45","link":"/2016/05/smartwatches-with-higher-bandwidth-vibration-notifications/","section":"post","summary":"This is an updated version of an article first published on Medium on Oct. 24, 2015.\nI love my smartwatch, way more than I thought I would when I bought it, over a year ago. It’s a Moto 360, which is …","tags":["gadgets","smartwatches","technology"],"title":"Smartwatches with Higher-Bandwidth Vibration Notifications"},{"body":" This post was originally published on Medium\nView this post on X (josh_wills) There are different types of data scientists, with different backgrounds and career paths. With Sean Murphy and Marck Vaisman, I wrote an article about this for O’Reilly a few years back, based on survey research we’d done. Download a copy, if you haven’t read it. This idea is now pretty well established, but I want to talk about a related issue, which is that the type of workthat Data Science teams do varies a lot, and that managing those types of work can be an interesting challenge.\nAs Josh Wills said, data scientists aren’t software developers, but they sometimes do that sort of work, and they aren’t statisticians, but they sometimes do that sort of work too. At EAB, where I lead a Data Science team of people with very diverse backgrounds and skill sets, this issue leads to a lot of complexity and experimentation as we (and the upper management I report to) try to ensure that everyone is working on the right tasks, at the right time, efficiently.\nIn this post, I’d like to share some thoughts about how we currently think about and manage different types of Data Science work. I also wrote a little Shiny web tool to help us manage our time, and I’ll show that off as well.\nA recent innovation on our team is partial adoption of Agile/Scrum development processes. We now have daily standup meetings, two-week sprints with kickoff and retrospective meetings, and time estimation for (some of) our tasks. In general, I’m not a fan of process for its own sake, but as the team has grown, it’s become more important to have visibility into projects, to ensure that they’re on-track and not stuck or moving off on a (fascinating) tangent. So far, two of the most valuable pieces of Scrum are having processes that cause us to add more definition to projects and tasks, along with the retrospective meeting, where we review the most recent sprint and try to improve our own processes.\nScrum has limitations for us, though. For one thing, most of our projects are fairly specialized, and the standard practice of having the team as a whole estimate task difficulty is not viable when backgrounds differ so much. So we have the people who are working on a project or task estimate for themselves.\nThe bigger issue is the variety of workstreams. For us, we have four main categories of work:\nDevelopment– writing code for applications, either analytic services that are part of our company’s product, or internal web applications used by us or others. All of us do this work, but some data scientists specialize in this sort of work.\nResearch– trying to understand something better. We divide this into three subcategories:insights research is about understanding our customers’ domain and data better;algorithm research is about matching data science approaches to business needs; technology research is about improving our technology stack.\nService– repeatable operations tasks, usually customer specific, such as model-fitting or ad hoc analysis.\nTeam Development– interviewing, training, teaching, writing blog posts, volunteer work, etc.\nOur Development work is not that different from our colleagues who sling Java all day (except that we don’t have to sling Java, thankfully). We use relatively standard Scrum processes for these tasks, including breaking down Epics into smaller Tasks, estimating those Tasks, having Task kick-off meetings to ensure requirements are clear, and of course we use modern software development processes such as git, code reviews, and continuous testing and integration.\nOur Research work is a hybrid. For a six-week project, we might have a high-level project plan that keeps the work focused on specific questions to be answered, as well as a series of technical tasks that are managed mostly like Development work. If you’re trying to figure out what algorithm to use for something, you probably have tasks that look like “read papers” as well as tasks that look like “build data pipeline”. In the former case we time-box the work, to ensure that it’s targeted and not encyclopedic; in the latter case, we usually have enough information to treat the task as Development work once it’s ready to start.\nOur Service work is totally different. It’s often kicked off and constrained by other teams, and we manage it with a fairly traditional Kanban process. People who have time pick up a ticket and own until it’s done. (Our primary Service tasks typically take 4 to 8 hours of work, spread out over a couple of weeks as we run into issues, resolve them, get feedback, and so on.) We track velocity, but don’t estimate it.\nFinally, our Team Development work is even more ad hoc, and we don’t currently do any sort of task tracking here, aside from conversations at one-on-one meetings.\nA few months back, it became clear that allocating people’s time between these workstreams was really important. Even if we were managing specific projects well, we weren’t necessarily balancing them at a high level. Because of this, we now allocate each person’s time (including mine) into the four workstreams for each two-week sprint. This helps keep people on task, and even more importantly, keeps them from taking on too much of one type of work and neglecting other projects, without having to micromanage every task or treat all tasks the same.\nTo manage and communicate this time allocation, I developed an incredibly quick-and-dirty Shiny web application called Team Time. (I’ve spent more time writing this blog post than I did on the first version of the tool!) Team Time does one thing — it manages a table of people, workstreams, and days, allows interactive editing, and shows handy summary graphs.\nAs you can see in the screenshot, the current version has two tables, Current and Next, so you can work on time allocation for the next sprint while you keep the table available for the current one. You can add rows (but not columns, for technical reasons) from the UI, and you can edit cells in the table. When you press Save, the table is saved and other people will see the updates when they reload.\nWhy is this useful? Nearly identical functionality could be built as a shared Excel or Google Sheets document. But UX matters, and many companies have grown from nothing by taking an Excel template and turning it into an easy-to-use web app. In this case, we get a tool with a simple, functional interface that does what we need, written with tools we know, and it’s accessible in a single click from our team’s web portal.\nHow do we use it? During our biweekly sprint kickoffs, we walk through Next, make changes from my proposed time allocation as necessary, Save it to Current, then move on to more detailed assignment of tasks to the new sprint. The total days per area give us valuable guidelines for how many points of tasks from the Development and Research backlogs should be moved into the sprint. And having this high-level view of workstreams helps people manage expectations, and provides a backdrop for retrospective meetings as well.\nOver time, as we learn more, I expect the tool to continue to evolve. Better history might be handy, for reporting purposes. Synchronized changes would be fun. Maybe Zawinski’s Law will apply, and it’ll send email at some point.\nAs this tool is not a core competency of our company, I’ve open sourced the application, so you can use it (or improve it), if you’d like!\nA sample, public version of the app is on ShinyApps. Feel free to play around with it. Right-click on the table to add/delete rows. (Note that whenever the app restarts, which it might do at arbitrary times, the tables on the disk is reset. Our copy is running inside our firewall on a copy of Shiny Server.)\nThe code is on GitHub, MIT licensed. A few shout-outs about handy R packages that made this possible with almost no effort. Shiny, of course, for the web app framework, and shinydashboard for the template. rhandsontable, an R wrapper for the handsontable.js Javascript library, for the interactive table. And ggplot2, magrittr, and tidyr from the Hadleyverse made the table manipulation and visualization almost trivial. Pull requests very welcome!\nOh, and if you’ll be at the Shiny Developers Conference this weekend (Jan 30th-31st, 2016), and want to chat about any of this, see you there!\n","date":1453939200,"id":"46","link":"/2016/01/thoughts-on-managing-data-science-team-workstreams-and-a-shiny-app/","section":"post","summary":"This post was originally published on Medium\nView this post on X (josh_wills) There are different types of data scientists, with different backgrounds and career paths. With Sean Murphy and Marck …","tags":["data science","management","business","R","Shiny"],"title":"Thoughts on Managing Data Science Team Workstreams (and a Shiny app)"},{"body":" This post was originally published on Medium\nI’m the Director of Data Science at EAB, a firm that provides best-practices research and enterprise software for colleges and universities. My team is responsible for the predictive models and other advanced analytics that are part of the Student Success Collaborative product that’s used by academic advisors and other campus leadership. We’re hiring data scientists, and I wanted to publicly say a few things about the roles we have advertised. (Note that EAB is part of a public company and is in a competitive market, so there are obviously things I’m not saying!)\nThe most important point is that data scientists specialize, so look for the specializations. My co-authors and I made this point in our 2012 e-book Analyzing the Analyzers, and the folks at Mango Solutions are burning up Twitter with their self-service tool for identifying data science strengths and weaknesses.\nDrew Conway’s Data Science Venn Diagram — Source.\nA related point is that existing framing devices can help you balance a team. Drew Conway’s Venn Diagram remains a great way to think about Data Science aptitude. Combine people with strengths in each part of the diagram, who know enough to collaborate effective and make each other stronger, and you don’t need a team of unicorns with 3 PhDs each.\nI suspect the details of the framing device are less important than the fact that you have one. It forces you to think about variety and complementary skills, and how people work together to solve problems and build systems.\nAt EAB, we have four career tracks for data scientists — Research, Engineering, Statistical Programming, and Management. Our new roles supplement our existing team by adding several new people, each with different capabilities and seniority.\nAt a Senior level, we’re looking for a Statistical Programmer-track person who is particularly strong in algorithm development and implementation, perhaps a straight-up Computer Scientist. Think of the “Machine Learning” area in Drew’s diagram. As we look to expand the classes of statistical techniques that we use, we need more people who know the academic literature and can figure out exactly what technical solution will let us build and scale high-quality models. Interested? Please apply!\nA little less senior, we’re also looking for a Researcher who can help us apply domain knowledge even more effectively in our analyses, models, and systems. Some software, data visualization, and statistical skills required — maybe a quantitative Social Scientist pivoted into industry? The upper edge of the Substantive Expertise area. Sound like you? Please apply!\nI strongly believe that a Data Science team should do all of the Data Science, including building and owning models in production. So, last but not least, we’re looking for another Engineering-focused data scientist, who can help us build model frameworks, data tools, workflow tools, and more. This role can be junior or even entry-level, but we do need programming skills, statistical thinking skills, and some sort of portfolio. Programmer and recent data science boot camp grad, perhaps? Please apply!\nOf course, as we talk to people, learn what they’re good at and excited about, and what they bring, we may end up with a different mix of skills. But regardless, they’ll cover the space of data scientists, will provide different perspectives and skills, and will help us own our own tools and systems so that we can move and learn quickly.\n","date":1448236800,"id":"47","link":"/2015/11/building-a-complementary-data-science-team/","section":"post","summary":"This post was originally published on Medium\nI’m the Director of Data Science at EAB, a firm that provides best-practices research and enterprise software for colleges and universities. My team is …","tags":["data science","management","business","job titles"],"title":"Building a Complementary Data Science Team"},{"body":"The below is a public version of a post originally posted on an internal blog at the Education Advisory Board (EAB), my current employer. We don’t yet have a public tech blog, but I got permission to edit and post it here, along with the referenced code. Data Science teams get asked to do a lot of different sorts of things. Some of what the team that I’m part of builds is enterprise-scale predictive analytics, such as the Student Risk Model that’s part of the Student Success Collaborative. That’s basically software development with a statistical twist and machine-learning core. Sometimes we get asked to do quick-and-dirty, one-off sorts of things, to answer a research question. We have a variety of tools and processes for that task. But there’s a third category that I want to focus on – frequently requested but slightly-different reports. what is it There’s a relatively new theme in the scientific research community called reproducible research. Briefly, the idea is that it should be possible to re-do all steps after data collection automatically, including data cleaning and reformatting, statistical analyses, and even the actual generation of a camera-ready report with charts, graphs, and tables. This means that if you realized that, say, one data point in your analysis was bogus and needed to be removed, you could remove that data point, press a button, and in a minute or two have a shiny new PDF with all of the results automatically updated.\nThis type of reproducible research has been around for a while, although it’s having a recent resurgence in part due to the so-called “statistical crisis”. The R (and S) statistical programming languages have supported LaTeX, the scientific document creation/typesetting tool, for many years. Using a tool called Sweave, a researcher “weaves” chunks of text and chunks of R code together. The document is then “executed”, where the R code chunks are executed and the results are converted into a single LaTeX document, which is then compiled into a PDF or similar. The code can generate charts and tables, so no manual effort is needed to rebuild a camera-ready document.\nThis is great, a huge step forward towards validation of often tricky and complex statistical analyses. If you’re writing a conference paper on, say, a biomedical experiment, a reproducible process can drastically improve your ability to be confident in your work. But data scientists often have to generate this sort of thing repeatedly, from different sources of data or with different parameters. And they have to do so efficiently.\nParameterizable reproducible research, then, is a variant of reproducible research tools and workflows where it is easy to specify data sources, options, and parameters to a standardized analytical report, even one that includes statistical or predictive analyses, data manipulation, and graph generation. The report can be emailed or otherwise sent to people, and doesn’t seem as public as, say, a web-based app developed in Shiny or another technology. This isn’t a huge breakthrough or anything, but it’s a useful pattern that seems worth sharing.\nwhy do it Our current best example of this structure is a report we generate that goes into per-customer technical details about their customized statistical model. It compares the version of the model in production with several “toy” models, along with a lot of explanation and education. The process of generating the report includes pulling data from remote servers, running statistical analyses, computing various metrics on the statistical models, then interleaving standard text, parameterized text (e.g., the name of the customer), and the results of computation, as well as auto-generated charts and graphs.\nTo do this process manually would require following a 10 to 20-step checklist, would require a substantial amount of effort – at least 30 minutes per report – and would be error-prone.\nOur team now generates these reports in about 3 minutes per member, based on a simple configuration file. Here’s what a configuration file might look like:\nuser = \u0026quot;hharris\u0026quot; data_source = \u0026quot;SOURCENAMEHERE\u0026quot; data_location = \u0026quot;where in the source to pull data from\u0026quot; cust_id = \u0026quot;9999\u0026quot; cust_name = \u0026quot;State College\u0026quot; That’s it. And here’s the command that generates the report:\n./risk-model-report.R --verbose --config myconfig.R That’s it! The result is a standalone HTML file customized for that specific customer, ready to be sent.\nShould you use this pattern? If you use R to create reports, you should definitely be using Rmarkdown. If you have to generate these reports repeatedly, with subtle variations each time, you should strongly consider this framework or an equivalent! how we do it We use a relatively straightforward pattern available in the R programming language. As mentioned above, the standard reproducible research workflow is to create one document per analysis. We wrap that document in a separate script which is responsible for reading and validating a configuration file, then building the parameterized document with the appropriate configuration variables. The build process is multi-step under the hood, but most of the heavy lifting is performed by the Rmarkdown package, which runs Pandoc, a cross-platform system for converting document formats.\nFor an example, see this HTML document (it should work in any web browser). Note that the image is embedded, meaning that the HTML document stands alone. It’s relatively easy to generate other formats, such as PDF or even Word, instead.\nThe document was generated by R and Rmarkdown code we’ve released into the public domain, hosted on Github. If this pattern is useful to you, please make use of and adapt it!\nNote that since the generated code is HTML, you can add arbitrary Javascript and markup to your generated reports. We’ve used collapsible areas of a page to expose technical details to only interested readers, added MathJax for mathematical expressions, and even added Javascript libraries to send us data about how many times the document is viewed and whether outgoing links were followed.\nOver time, we expect to use variants of this pattern to standardize a variety of reports, internally-facing as well as customer-facing. If you do something similar, or better, I’d love to hear about it!\n","date":1416441600,"id":"48","link":"/2014/11/parameterizable-reproducible-research/","section":"post","summary":"The below is a public version of a post originally posted on an internal blog at the Education Advisory Board (EAB), my current employer. We don’t yet have a public tech blog, but I got permission to …","tags":["advertising","programming","R","reporting"],"title":"Parameterizable Reproducible Research"},{"body":" View this post on X (HarlanH) Let me unpack that a bit…\nHugh and Crye t-shirt\nRecently, Hugh \u0026amp; Crye, a DC-based clothing firm for men, with a novel take on sizing, recently did a Kickstarter campaign for their new line of fitted t-shirts. What the hell? H\u0026amp;C has been around for about 5 years, and based on their product growth and hiring seems to be doing quite well. I like their stuff. Why do they need a Kickstarter? The original goal of Kickstarter was to “kickstart” new products by providing crowdsourced seed funding so that you (you!) can ensure that a great idea gets off the ground. And if a project doesn’t make its goals, no harm done, and no money wasted. A fantastic example is the Oculus Rift, which was a Kickstarted Virtual Reality rig, and is now a subsidiary of Facebook. Kickstarting a project is a rather labor-intensive alternative to trying to get a bank loan, or maxing out your credit cards, but with much less risk. It’s a very community-driven, authentic way of getting support for a new venture, moving it from the prototype phase to the initial manufacturing round.\nDoes any of this apply to Hugh \u0026amp; Crye? Do they need free money from the community? No, they’re clearly cash-flow positive and could get a bank loan. Do they need help figuring out manufacturing? No, clearly not. Was there any doubt that their loyal customers would fail to buy t-shirts, where the all-or-nothing nature of a Kickstarter would provide them with better understanding of the market? Heck no – they’ve been doing market research on this for years. Will the amount of money they get actually help their operations move faster? They made $53,151 – probably not.\nSo why are they doing it? To drum up buzz – that’s it. It’s advertising for a done-deal. Kickstarter is not a platform for advertising. The campaign is inauthentic and annoying.\n(But I still supported it, because I wanted to try one of their t-shirts.)\nChick-fil-A sandwich\nHighly related issue: food trucks from big companies. Chick-fil-A, the chicken sandwich chain, has food trucks that have drawn lots of controversy. Some of that is because of the company’s politics. But if you read comments on Facebook and Twitter, a fair amount of the criticism is because they’re inauthentic – that they’re cheapening the authentic experience of trying someone’s labor of love. That they’re insulting your hope that your favorite food truck will be successful enough to open a sit-down restaurant.\nIt’s inauthentic and annoying.\nDo you have other examples that get your goat? Please leave a comment!\nTo close, for a great take on authenticity, you should definitely watch this great video from my new favorite YouTube Channel, the boringly-named but completely awesome PBS Idea Channel:\n","date":1410048e3,"id":"49","link":"/2014/09/inauthenticity/","section":"post","summary":" View this post on X (HarlanH) Let me unpack that a bit…\nHugh and Crye t-shirt\nRecently, Hugh \u0026amp; Crye, a DC-based clothing firm for men, with a novel take on sizing, recently did a Kickstarter …","tags":["advertising","marketing","business","media"],"title":"Inauthenticity"},{"body":" Earlier this year, I attended the INFORMS Conference on Business Analytics \u0026amp; Operations Research, in Boston. I was asked beforehand if I wanted to be a conference blogger, and for some reason I said I would. This meant I was able to publish posts on the conference’s WordPress web site, and was also obliged to do so!\nHere are the five posts that I wrote, along with an excerpt from each. Please click through to read the full pieces: Operations Research, from the point of view of Data Science more insight, less action — deliverables tend towards predictions and storytelling, versus formal optimization more openness, less big iron — open source software leads to a low-cost, highly flexible approach more scruffy, less neat — data science technologies often come from black-box statistical models, vs. domain-based theory more velocity, smaller projects — a hundred $10K projects beats one $1M project more science, less engineering — both practitioners and methods have different backgrounds more hipsters, less suits — stronger connections to the tech industry than to the boardroom more rockstars, less teams — one person can now (roughly) do everything, in simple cases, for better or worse What is a “Data Product”? DJ Patil says “a data product is a product that facilitates an end goal through the use of data.” So, it’s not just an analysis, or a recommendation to executives, or an insight that leads to an improvement to a business process. It’s a visible component of a system. LinkedIn’s People You May Know is viewed by many millions of customers, and it’s based on the complex interactions of the customers themselves. Healthcare (and not Education) at INFORMS Analytics [A]s a DC resident, we often hear of “Healthcare and Education” as a linked pair of industries. Both are systems focused on social good, with intertwined government, nonprofit, and for-profit entities, highly distributed management, and (reportedly) huge opportunities for improvement. Aside from MIT Leaders for Global Operations winning the Smith Prize (and a number of shoutouts to academic partners and mentors), there was not a peep from the education sector at tonight’s awards ceremony. Is education, and particularly K-12 and postsecondary education, not amenable to OR techniques or solutions? What’s Changed at the Practice/Analytics Conference? In 2011, almost every talk seemed to me to be from a Fortune 500 company, or a large nonprofit, or a consulting firm advising a Fortune 500 company or a large nonprofit. Entrepeneurship around analytics was barely to be seen. This year, there are at least a few talks about Hadoop and iPhone apps and more. Has the cost of deploying advanced analytics substantially dropped? Why OR/Analytics People Need to Know About Database Technology It’s worthwhile learning a bit about databases, even if you have no decision-making authority in your organization, and don’t feel like becoming a database administrator (good call). But by getting involved early in the data-collection process, when IT folks are sitting around a table arguing about platform questions, you can get a word in occasionally about the things that matter for analytics — collecting all the data, storing it in a way friendly to later analytics, and so forth. All in all, I enjoyed blogging the conference, and recommend the practice to others! It’s a great way to organize your thoughts and to summarize and synthesize your experiences.\n","date":1406937600,"id":"50","link":"/2014/08/informs-business-analytics-2014-blog-posts/","section":"post","summary":"Earlier this year, I attended the INFORMS Conference on Business Analytics \u0026amp; Operations Research, in Boston. I was asked beforehand if I wanted to be a conference blogger, and for some reason I …","tags":["analytics","data science","operations research"],"title":"INFORMS Business Analytics 2014 Blog Posts"},{"body":" On Monday, October 28th, 2013, I gave a 5-minute Ignite talk entitled “Why a Data Community is Like a Music Scene” at an event associated with the Strata conference. Here’s the video:\nAnd here are the acknowledgements and references for the talk:\nData Community DC\nHow Music Works, by David Byrne\nmy slides for the Ignite talk\nmy blog post (written first)\nPhotos:\nCBGB’s exterior: NYC - East Village: CBGB \u0026amp; OMFUG by wallyg, on Flickr (Creative Commons) Grafitti wall: cbgb, september 2006 (#1) by joe holmes, on Flickr (Creative Commons) Joan Jett with beer: 1977 Los Angeles, CA-Joan Jett of the Runaways backstage at The Whiskey A Go Go. (Photos by Brad Elterman/BuzzFoto.com) CSN: legendsrevealed.com Flats to Let: skyscrapercity.com Dressing room: CBGBS Dresssing Room by Chris Infidel, on Flickr Snog: _MG_1977.jpg by dinoboy, on Flickr ","date":1382745600,"id":"51","link":"/2013/10/why-a-data-community-is-like-a-music-scene-resources/","section":"post","summary":"On Monday, October 28th, 2013, I gave a 5-minute Ignite talk entitled “Why a Data Community is Like a Music Scene” at an event associated with the Strata conference. Here’s the video:\nAnd here are the …","tags":["conferences","data science"],"title":"Why a Data Community is Like a Music Scene -- Resources"},{"body":" A topographic map of Washington in 1791 by Don Alexander Hawkins. I live on the top edge of the map, on one of those hills.\nI’m a generally happy user of DC’s Capital Bikeshare system – just renewed my annual membership today in fact. But I don’t use it as much as I’d like to, for one critical reason. I live on top of a hill. Riders are happy to take bikes from the neighborhood to their jobs downhill, but are much less likely to ride them uphill. As a result, the bike racks in my neighborhood are frequently completely empty by 8:00 or 8:30am, despite the many stores and businesses in the area. The only days I can reliably take a bike into work are when I leave at 7:15 for 8:00 meetings, which is thankfully not too often. On several occasions I have looked at the handy real-time map of bikeshare bikes, only to observe that there are no bikes available within a 15 minutes walk of my home!\nWhat should Bikeshare do to solve this problem? Well, they already do one thing, which is that they hire people to put bicycles in the back of a big van, then drive them up the hill to rebalance the system. This works, but it’s expensive for the system, and it’s not very timely or efficient. In other transportation problems, incentives are used to balance demand. For instance, airlines and Amtrak use pricing to incentivize people who are flexible in their schedule to take off-peak trips. But that won’t work for Bikeshare, as most rides are free. (I pay $75/year, but all trips of 30 minutes or less are free. My rides are mostly 15-25 minutes long.) So people happily ride downhill to their downtown jobs in the mornings, but don’t ride uphill to their reverse-commute jobs, and don’t as often ride uphill in the evening home either. The end result is unhappy customers and excessive costs for the Bikeshare system.\nRough map of a possible incentive line for North-Central DC.\nSo if you can’t give people the usual financial incentives to drop off bikes in the Columbia Heights rack at 8am, what can you do to reduce the need for rebalancing and provide reasons for people to want to help solve Bikeshare’s problem? I think the answer is swag. Imagine that there were lines on the Bikeshare map. Every time you crossed the line going in an uphill direction (reducing the need for rebalancing), you’d earn some points. If you earned enough points, you could redeem them for Bikeshare-branded, limited-edition swag. Imagine a t-shirt in official CaBi colors that said “I bike up hills”, available only through this point system. Who wouldn’t want that?\nIt’s easy for Bikeshare to figure this out, as they know exactly where you picked up each bike, and where you dropped it off. Determining whether you crossed a line, and thus biked uphill, is easy. And in addition to making people excited about biking up hills, you get them wearing branded items of clothing, which can only help market the system more broadly. They already sell swag through a Cafepress shop, so much of the infrastructure is in place. It’s a win-win.\nBikeshare people, if you read this and think it’s a good idea, please run with it!\nWhat the swag might look like.\n","date":1371945600,"id":"52","link":"/2013/06/bikeshare-hills-incentives-and-rewards/","section":"post","summary":" A topographic map of Washington in 1791 by Don Alexander Hawkins. I live on the top edge of the map, on one of those hills.\nI’m a generally happy user of DC’s Capital Bikeshare system – just renewed …","tags":["bikeshare","marketing","optimization"],"title":"Bikeshare hills, incentives, and rewards"},{"body":" And, we’re back! After being off-line for several weeks, this site is now live again! I can’t imagine you missed it.\nHere’s what happened. Let’s start at the beginning. In 2003, ICANN added .name to the list of top-level domains (like .com, .edu, etc.). The idea is that individuals would use it for personal sites and email addresses. You can still do this, but (in case you haven’t noticed), it’s not very popular, and most domain name registrars don’t even sell .name addresses.\nI purchased harlan.harris.name in 2003. Unlike .com addresses, you don’t generally buy second-level domains in .name, you buy third-level domains. (.name is the top-level domain, harris.name is the second-level domain, which you can’t buy, and harlan.harris.name is the third-level domain.) A cool feature is that if you buy a.b.name, you can get the email address a@b.name, not something like me@a.b.name (although you can set that up too). So my email address has been harlan@harris.name for ten years.\nFast forward to April, 2013. I notice that my personal web site (where you are now) has been replaced by a generic sales screen. You know, with a bunch of random keywords, a stock photo, and “buy this domain!” in big red print. Not good. At first I thought that my Wordpress site (which hosts this blog) had gotten hacked, but no such luck. It turns out to be a convoluted mess of broken technology and confused customer support reps. The fortunate thing is that I don’t use this site extensively, and the problem with the web forwarding didn’t seem to affect my email address forwarding, so I didn’t lose any email.\nThe simplified version of what happened is that the company I bought the domain from in 2003, PersonalNames, merged with a company called Dotster a year or two ago. They presumably merged their technical systems together, which makes sense. But they for some reason failed to properly set up a system for third-level .name domain administration. And so my account failed to get properly transferred into their systems, and they stopped sending me notices about problems.\nAlthough I still technically owned harlan.harris.name, I could no longer log in and administer it, and the redirection to this web site (at another company, HostGator) was reset at some point for still-unknown reasons.\nIt took a week and a dozen email messages and several hours on the phone for Dotster to figure out that yes, they owned this domain, but no, they didn’t have the technical chops to administer it.\nI then set up an account with another company, eNom (nom, nom…), that does support third-level .name domains. Transferring the domain took another week and three attempts, due to errors on both sides. Add 48 hours for DNS forwarding to propagate around the Internet, and I’m finally back online yesterday!\nExcept that although my email forwarding still works, I don’t yet have control over that, because Dotster seemingly neglected to transfer email forwarding rights at the same time as the rest of the domain. So if you need me tomorrow, I’ll be back on the phone with tech support.\n","date":1368576e3,"id":"53","link":"/2013/05/on-name-and-third-level-domains/","section":"post","summary":"And, we’re back! After being off-line for several weeks, this site is now live again! I can’t imagine you missed it.\nHere’s what happened. Let’s start at the beginning. In 2003, ICANN added .name to …","tags":["internet","meta"],"title":"On .name and third-level domains"},{"body":" For those people (or, more likely, 0 or 1 persons) who follow this blog to catch up on my professional thoughts: I’ve been doing a little bit of writing on the Data Community DC blog. Here are all my posts over there: http://datacommunitydc.org/blog/author/harlan/ I’d definitely encourage you to read everyone else’s work on the DC2 blog too!\nTwo titles of my own:\nExamining Overlapping Meetup Memberships with Venn Diagrams Hackathons and DataDives And three of others’:\nBig Data Meta Review – The Evolution of Big Data as Told by Google Better Science of Viral Marketing Getting Started with Recommendation Systems There are also weekly round-up posts on data topics generally, and on data visualization specifically, as well as event previews and reviews, etc.\n","date":1361404800,"id":"54","link":"/2013/02/more-posts-on-the-data-community-dc-blog/","section":"post","summary":"For those people (or, more likely, 0 or 1 persons) who follow this blog to catch up on my professional thoughts: I’ve been doing a little bit of writing on the Data Community DC blog. Here are all my …","tags":["data science","meta"],"title":"More posts on the Data Community DC blog"},{"body":" I recently gave a presentation on communication issues around the terms “Data Science” and “Data Scientist”, based in part on a survey that I did with my Meetup colleagues Marck and Sean. The basic idea is that these new, extremely-broad buzzwords have resulted in confusion, which has impacted the ability of people with skills and people with data to meet and effectively communicate about who does what and what appropriate expectations should be. The survey was an attempt to bring some clarity to the issue of who are the people in this newly-reformulated community, and how do they view themselves and their skills. For more on the survey, see our post on the Data Community DC blog. Here’s the video of my presentation at DataGotham:\n","date":1350950400,"id":"55","link":"/2012/10/communication-and-the-data-scientist/","section":"post","summary":"I recently gave a presentation on communication issues around the terms “Data Science” and “Data Scientist”, based in part on a survey that I did with my Meetup colleagues Marck and Sean. The basic …","tags":["analytics","conferences","data science","survey","video"],"title":"Communication and the Data Scientist"},{"body":" My newish cooking club had a dinner yesterday with the theme American Beer. I was tasked with dessert, and came up with this recipe for Pretzel Whoopie Pies. They turned out extremely well, so I thought I’d share the recipe here.\nSources:\nAbout.com Southern Food (basic whoopie pie recipe) Southern Living (Stout buttercream) Ideas In Food (flavor combo) Bakewise (cookie recipe tweaks) Ingredients:\n2 egg yolks 1/2 c minus 1 T sugar 1 T light corn syrup 1/2 c finely ground unsalted mini pretzels 1/2 c cake flour 1 t baking powder 1/8 t salt 4 T butter, softened 1/3 c milk kosher salt 1 c stout beer (Breckenridge Vanilla Porter is excellent) 4 T butter, softened 8 oz powdered sugar Recipe:\nBeat egg yolks, sugar, and corn syrup until lightened. Mix pretzel flour, cake flour, baking powder, and salt. Beat in butter and milk. Add to liquid mixture and mix thoroughly. Refrigerate dough for 30 minutes to hydrate evenly. Preheat oven to 350 F. Drop 12 evenly-shaped cookies onto a silpat-covered pan. A ring mold is helpful. Sprinkle kosher salt over the top lightly. Bake about 14 minutes, until starting to brown around the edges, but still soft. Cool thoroughly on wire racks. Boil beer in a saucepan large enough to deal with foaming up, and reduce to 1/2 c. Cool to room temperature. Mix butter, powdered sugar, and reduced beer into a creamy, delicious frosting. Make sandwiches out of cookies and frosting. Makes 6 whoopie pies.\n","date":1350864e3,"id":"56","link":"/2012/10/pretzel-whoopie-pies-with-vanilla-stout-filling/","section":"post","summary":"My newish cooking club had a dinner yesterday with the theme American Beer. I was tasked with dessert, and came up with this recipe for Pretzel Whoopie Pies. They turned out extremely well, so I …","tags":["cooking club","recipes"],"title":"Pretzel Whoopie Pies with Vanilla Stout Filling"},{"body":" I just returned from the useR! 2012 conference for developers and users of R. One of the common themes to many of the presentations was integration of R-based statistical systems with other systems, be they other programming languages, web systems, or enterprise data systems. Some highlights for me were an update to Rserve that includes 1-stop web services, and a presentation on ESB integration. Although I didn’t see it discussed, the new httr package for easier access to web services is also another outstanding development in integrating R into large-scale systems.\nCoincidentally, I just a week or so ago had given a short presentation to the local R Meetup entitled “Annotating Enterprise Data from an R Server.” The topic for the evening was “R in the Enterprise,” and others talked about generating large, automated reports with knitr, and using RPy2 to integrate R into a Python-based web system. I talked about my experiences building and deploying a predictive system, using the corporate database as the common link. Here are the slides:\n","date":1339804800,"id":"57","link":"/2012/06/integrating-r-with-other-systems/","section":"post","summary":"I just returned from the useR! 2012 conference for developers and users of R. One of the common themes to many of the presentations was integration of R-based statistical systems with other systems, …","tags":["analytics","conferences","data science","meetup","programming","R"],"title":"integrating R with other systems"},{"body":" As I’ve discussed here before, there is a debate raging (ok, maybe not raging) about terms such as “data science”, “analytics”, “data mining”, and “big data”. What do they mean, how do they overlap, and perhaps most importantly, who are the people who work in these fields?\nAlong with two other DC-area Data Scientists, Marck Vaisman and Sean Murphy, I’ve put together a survey to explore some of these issues. Help us quantitatively understand the space of data-related skills and careers by participating!\nSurvey link: http://bit.ly/IQNM5A\nIt should take 10 minutes or less, data will be kept confidential, and we look forward to sharing our results and insights in a variety of venues, including this blog! Thanks!\n","date":1336608e3,"id":"58","link":"/2012/05/survey-of-data-science-analytics-big-data-applied-stats-machine-learning-etc-practitioners/","section":"post","summary":"As I’ve discussed here before, there is a debate raging (ok, maybe not raging) about terms such as “data science”, “analytics”, “data mining”, and “big data”. What do they mean, how do they overlap, …","tags":["analytics","data science","meta","operations research","R","survey"],"title":"Survey of Data Science / Analytics / Big Data / Applied Stats / Machine Learning etc. Practitioners"},{"body":"On last week’s Build and Analyze – a great podcast nominally about iOS development, but actually more about just living a tech-geek lifestyle – Marco talked a lot about the rumored “Apple TV” and whether it could actually be a groundbreaking product. He concluded that it probably couldn’t. Most people wouldn’t dump a working TV just for an Apple brand; the touch-screen interface that Apple has been using for the iPad and iPhone wouldn’t work for a TV; the only apps that would work well on a TV would be just ways of getting better content (I note that Roku apps are laughable, with the exception of Angry Birds); getting access to better content than other competitors is probably impossible, even for Apple.\nFor these reasons and more, Marco suggested that there’s little that Apple, or anyone else, could do to substantially improve the TV experience, with the exception of better menu design.\nI think there’s a way that Apple (or someone) could integrate modern technology into a TV that would be actually compelling, though. And in some ways it’s the same way that I earlier blogged about for MP3 players. Cross-device user-interfaces. Here’s how it might work for a television:\nImagine a shiny Apple-branded TV. It’s got a power cord, an Ethernet jack, and audio out jacks, full stop. The TV has a built-in Wi-Fi hub (AirPort), so you don’t need another one.\nIt doesn’t have a dedicated remote control, but instead it uses your iPhone or iPod Touch or iPad. (Read all this as “Google TV” and “Android phone”, if you prefer.) And it’s the best remote control you’ve ever seen. Wi-fi fast and no need to point it at your screen. It has natural, context-dependent, easy-to-use controls, making use of multitouch and other gestures. A TV can have an amazing touch interface if you’re holding one of the screens!\nAnd content will be almost magically flexible. Got a movie on your iPhone you were watching on the subway? It streams wirelessly to your TV when you get home. Watching a movie on TV but have to move to the other room? Now it’s streaming to your tablet. Watching something with multiple views, like sports? Picture-in-picture is so ’90s, it’s two screens now. In fact, you can easily switch any content from one screen to another, or use one device for audio and one for video of the same content.\nYour TV doesn’t feel like a separate appliance anymore, it’s an extension of your handheld device, with both connected to the cloud. Apps can make use of both screens (and both CPUs), separating controls from content, or primary from secondary displays in games, or any number of other things.\nImagine how much simpler and less frustrating this experience could be! Somebody make this happen!\n","date":1320537600,"id":"59","link":"/2011/11/apple-tv-and-cross-device-user-interface-integration/","section":"post","summary":"On last week’s Build and Analyze – a great podcast nominally about iOS development, but actually more about just living a tech-geek lifestyle – Marco talked a lot about the rumored “Apple TV” and …","tags":["gadgets","smartphones","technology","televisions","ui integration"],"title":"Apple TV and cross-device user-interface integration"},{"body":" I’m fond of navel gazing, meta discussions, and so forth. I’ve recently written about inferring navel gazing from link data, and about the meaning of the “Analytics” buzzword. This post will be my second on that other infectious buzzword, “Data Science”.\nWhen I moved to Washington DC in July, I was struck by the fact that there was no Meetup for analytics/applied statistics/machine learning/data science. There’s a great DC Tech Meetup, a great Big Data Meetup, and a great R Meetup, but nothing like the NYC Predictive Analytics Meetup. So, I and a couple of others I talked to about this (Marck Vaisman, who I first met through the NYC R Meetup a couple years ago, and Matt Bryan, who I met just after moving to town), started a new Meetup, which we decided to call “Data Science DC”.\nFor our second meetup, we thought we should address some aspect of our name, and so I presented a little bit about the term and the controversies around its definition and its recent dramatic upsurge in popularity. Here are the slides (note that you should be able to click through the links on the slide to the source documents):\nI mostly didn’t present a personal opinion about what I though the term means, or what it should mean, but instead wanted to present a bunch of other peoples’ points of view to kick off an interesting discussion. And in that sense I succeeded. We had an exceedingly interesting conversation following my slides, and I think a couple of the most interesting ideas from the evening came out of that discussion.\nHere are three theses I’d like to propose.\n“Data Science” is defined as what “Data Scientists” do. What Data Scientists do has been very well covered, and it runs the gamut from data collection and munging, through application of statistics and machine learning and related techniques, to interpretation, communication, and visualization of the results. Who Data Scientists are may be the more fundamental question. One reason Data Science is a big thing now is because advances in technology have made it easy for Data Scientists to develop wide-ranging expertise. Even 10 years ago, the idea that the same person could integrate several databases, run a multilevel regression, and generate elegant visualizations would be seen as incredibly rare. The other reason Data Science is a big thing now is because sabermetrics demonstrated that number-crunching brings results. There’s nothing business leaders love more than a sports analogy, and the analytic revolution in professional sports immediately draw attention to the ways that numbers beat intuition. I tend to like the idea that Data Science is defined by its practitioners, that it’s a career path rather than a category of activities. In my conversations with people, it seems that people who consider themselves Data Scientists typically have eclectic career paths, that might in some ways seem not to make much sense. A typical path might be someone who started out learning to program, then spent some time in a scientific field, then hopped around a variety of different roles, collecting a wide variety of different skills, all of which related to using analytical techniques to make sense of data.\nThis sort of career path isn’t particularly new, but what is new is that it’s now possible to relatively quickly and cheaply do get started in all of the processes involved in Data Science. (Thanks to Taylor Horton for suggesting this at the Meetup!) Fast computers, open source tools, and some programming skills allow someone to try a new data management approach or a new machine learning technology incredibly quickly, and to iterate on approaches until a solution to a particular problem is found. This has two consequences. First of all, the productivity of a modern Data Scientist is remarkable. Projects that a few decades ago would have taken teams of people literally years can now be done in a few days. Second of all, this amazing productivity allows people to spend their 10,000 hours developing expertise in the now vertically integrated process of Data Science, rather than having to spend all of that time focusing on developing skills on just a single aspect of the task. There are huge number of things that need to be learned to be an effective Data Scientist, but it is now possible to learn those skills quickly enough to make a career out of being a Jack of all Trades and a near-master at many of them.\nSo now there’s a supply of people who could be Data Scientists. But what about the type of demand that drives an incessant stream of O’Reilly articles and job postings? Where does the demand come from? Justin Grimes had an intriguing idea that resonated with me – analytics in sports, which I propose as the other reason why analytics and data science have become buzzwords. Although business has used mathematical methods for 100 years, (thousands if you include finance and insurance) the idea that you could hire a very small number of people to analyze data and beat gut instincts in many aspects of decision making is much newer. The idea that a statistician could turn around the Oakland A’s by radically overturning longstanding recruiting practice was a powerful analogy. Even now, business books about analytics almost always have sports examples in the first chapters. I made a point at work the other day by noting that most professional sports prognosticators predict NFL playoff outcomes wrong because they over-weight last years’ results. Sports analogies get attention.\nDoes this make sense? Data Science is a buzzword now because a group of people with eclectic talents match a growing demand for and recognition of the value of those talents. I’d love feedback on these thoughts!\n","date":1317081600,"id":"60","link":"/2011/09/data-science-moore-s-law-and-moneyball/","section":"post","summary":"I’m fond of navel gazing, meta discussions, and so forth. I’ve recently written about inferring navel gazing from link data, and about the meaning of the “Analytics” buzzword. This post will be my …","tags":["analytics","data science","meetup"],"title":"Data Science, Moore's Law, and Moneyball"},{"body":" This past Friday, the web portal to the US Federal government, USA.gov, organized hackathons across the US for programmers and data scientists to work with and analyze the data from their link-shortening service. It turns out that if you shorten a web link with bit.ly, the shortened link looks like 1.usa.gov/V6NpL (that one goes to a NASA page). And because this service was paid for by taxpayer money, the data about each clickthrough is freely available.\nShortened-link click-through data is interesting. It tells you the time and approximate geographic location of each click-through, and the web page or service that the link was on (assuming someone didn’t type the URL in by hand). You also know when the shortened link was created, which tells you a little bit about the way links are shared. Bit.ly themselves have several full time data scientists on staff whose job is to learn about what shortened-link data can say about web traffic patterns and link sharing, potentially very lucrative information.\nFor my part, I just wanted to do some fun visualizations. Along with friends in NYC, I joined the hackathon remotely, following along on twitter and listening to dance music in their turntable.fm room. I managed to get rough drafts of two somewhat non-trivial graphs done during the official hackathon, and I re-built them with larger and more random data later.\nThis first graph looks at the difference in time between when a link was created (the first time someone tried to shorten the target URL) and when the clickthroughs happened. For each of the 25 most frequently visited target domains (mostly US government agencies), I built a density plot, or smoothed histogram, of the timings.\nThere are some interesting differences. Links from senate.gov are mostly clicked through within a few hours of their creation, and links from the NY Courts are clicked through in less than an hour. There appear to be links to NOAA and the State of California pages that are frequently clicked through hundreds of days after their creation. It would be interesting to dive into the content of the target pages, categorize them, and learn what causes these differences.\nSpeaking of diving into the content, I did a very simple version of that next. When clicking a link to a government web page, are people looking for information about their hometown? Fortunately, clickthrough data includes geocode information for the clicker’s IP address, which includes the nearest city. I decided to find out by scraping the text content of the 100 most frequently accessed web pages, and detected whether or not each city was in each web page.\nThis “navel-gazers” plot shows the summarized results. For each city in the data set with more than 5 clickthroughs, I plotted the raw number of clickthroughs from that city (the X axis) against the proportion of clickthroughs that ended up on a web page with the name of the city in it (the Y axis). Many cities are clustered in the lower-left, with few clicks and no instances of their city on the target page. Large cities like New York and London are far to the right, as expected from their population, and they show up in target web pages occasionally. Washington (DC) is both a frequent clicker of shortened links, as well as a city that tends to show up on web pages, unsurprising given that it is the seat of the Federal government. The exceptions are the most interesting. People in Bangalore clicked through more than 15 times in this sample, and about 12% of their clicks were to pages with the name of their city. In Boulder, a quarter of the 12 or so clicks mentioned their town!\nDeeper analysis would be needed to explain these results, but they were fun to put together! For those interested in checking out my work, including R code to pull a sample of 1.usa.gov data from the archives, please check out my repository on GitHub: https://github.com/HarlanH/hackathon-1usagov\n","date":1311984e3,"id":"61","link":"/2011/07/hacking-gov-shortened-links/","section":"post","summary":"This past Friday, the web portal to the US Federal government, USA.gov, organized hackathons across the US for programmers and data scientists to work with and analyze the data from their …","tags":["analytics","dataviz","graphics","hackathon","R","visualization"],"title":"hacking .gov shortened links"},{"body":" In my previous post, I motivated a web application that would allow small-scale sustainable meat producers to sell directly to consumers using a meat share approach, using constrained optimization techniques to maximize utility for everyone involved. In this post, I’ll walk through some R code that I wrote to demonstrate the technique on a small scale.\nAlthough the problem is set up in R, the actual mathematical optimization is done by Symphony, an open-source mixed-integer solver that’s part of the COIN-OR project. (The problem of optimizing assignments, in this case of cuts of meat to people, is an integer planning problem, because the solution involves assigning either 0 or 1 of each cut to each person. More generally, linear programming and related optimization frameworks allow solving for real-numbered variables.) The RSymphony package allows problems set up in R to be solved by the C/C++ Symphony code with little hassle.\nMy code is in a public github repository called groupmeat-demo, and the demo code discussed here is in the subset_test.R file. (The other stuff in the repo is an unfinished version of a larger-scale demo with slightly more realistic data.)\nFor this toy problem, we want to optimally assign 6 items to 3 people, each of whom have a different utility (value) for each item. In this case, I’m ignoring any fixed utility, such as cost in dollars, but that could be added into the formulation. Additionally, assume that items #1 and #2 cannot both be assigned, as with pork loin and pork chops.\nThis sort of problem is fairly simple to define mathematically. To set up the problem in code, I’ll need to create some matrices that are used in the computation. Briefly, the goal is to maximize an objective expression, \\(\\mathbf{c}^T\\mathbf{x}\\), where the \\(\\mathbf{x}\\) are variables that will be 0 or 1, indicating an assignment or non-assignment, and the \\(\\mathbf{c}\\) is a coefficient vector representing the utilities of assigning each item to each person. Here, there are 6 items for 3 people, so I’ll have a 6x3 matrix, flattened to an 18-vector. The goal will be to find 0’s and 1’s for \\(\\mathbf{x}\\) that maximize the whole expression.\nHere’s what the \\(\\mathbf{c}\\) matrix looks like:\npers1 pers2 pers3 item1 0.467 0.221 0.2151 item2 0.030 0.252 0.4979 item3 0.019 0.033 0.0304 item4 0.043 0.348 0.0158 item5 0.414 0.050 0.0096 item6 0.029 0.095 0.2311 It appears as if everyone like item1, but only person1 likes item5.\nAdditionally, I need to define some constraints. For starters, it makes no sense to assign an item to more than one person. So, for each row of that matrix, the sum of the variables (not the utilities) must be 1, or maybe 0 (if that item is not assigned). I’ll create a constraint matrix, where each row contains 18 columns, and the pattern of 0’s and 1’s defines a row of the assignment matrix. Since there are 6 items, there are 6 rows (for now). Each row needs to be less than or equal to one (I’ll tell the solver to use integers only later), so I also define vectors of inequality symbols and right-hand-sides.\n# for each item/row, enforce that the sum of indicators for its assignment are \u0026lt;= 1 mat \u0026lt;- laply(1:num.items, function(ii) { x \u0026lt;- mat.0; x[ii, ] \u0026lt;- 1; as.double(x) }) dir \u0026lt;- rep(\u0026#39;\u0026lt;=\u0026#39;, num.items) rhs \u0026lt;- rep(1, num.items) To add the loin/chops constraint, I need to add another row, specifying that the sum of the indicators for both rows now must be 1 or less as well.\n# for rows 1 and 2, enforce that the sum of indicators for their assignments are \u0026lt;= 1 mat \u0026lt;- rbind(mat, matrix(matrix(c(1, 1, rep(0, num.items-2)), nrow=num.items, ncol=num.pers), nrow=1)) dir \u0026lt;- c(dir, \u0026#39;\u0026lt;=\u0026#39;) rhs \u0026lt;- c(rhs, 1) Here’s what those matrices and vectors look like:\n\u0026gt; mat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 [1,] 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 [2,] 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 [3,] 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 [4,] 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 [5,] 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 [6,] 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 [7,] 1 1 0 0 0 0 1 1 0 0 0 0 1 1 0 0 0 0 \u0026gt; dir [1] \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026gt; rhs [1] 1 1 1 1 1 1 1 Finally, specify that the variables must be binary (0 or 1), and call SYMPHONY to solve the problem:\n# this is an IP problem, for now types \u0026lt;- rep(\u0026#39;B\u0026#39;, num.items * num.pers) max \u0026lt;- TRUE # maximizing utility soln \u0026lt;- Rsymphony_solve_LP(obj, mat, dir, rhs, types=types, max=max) And, with a bit of post-processing to recover matrices from vectors, here’s the result:\n$solution [1] 0 0 0 0 1 0 0 0 1 1 0 0 0 1 0 0 0 1 $objval [1] 1.52 $status TM_OPTIMAL_SOLUTION_FOUND 0 Person #1 got Items 5 worth 0.41 Person #2 got Items 3, 4 worth 0.38 Person #3 got Items 2, 6 worth 0.73 So that’s great. It found an optimal solution worth more than 50% more than the expected value of a random assignment. But there’s a problem. There’s no guarantee that everyone gets anything, and in this case, person #3 gets almost twice as much utility as person #2. Unfair! We need to enforce an additional constraint, that the difference between the maximum utility that any one person gets and the minimum utility that any one person gets is not too high. This is sometimes called a parity constraint. Adding parity constraints is a little tricky, but the basic idea here is to add two more variables to the 18 I’ve already defined. These variables are positive real numbers, and they are forced by constraints to be the maximum and minimum total utilities per person. In the objective function, then, they are weighted so that their difference is not to big. So, that expression becomes: \\(\\mathbf{c}^T\\mathbf{x} - \\lambda x_{19} - - \\lambda x^{20}\\). The first variable (the maximum utility of any person) is minimized, while the second variable is maximized. The \\(\\lambda\\) free parameter defines how much to trade off parity with total utility, and I’ll set it to 1 for now.\nFor the existing rows of the constraint matrix, these new variables get 0’s. But two more rows need to be added, per person, to force their values to be no bigger/smaller (and thus the same as) the maximum/minimum of any person’s assigned utility.\n# now for those upper and lower variables # \\forall p, \\sum_i u_i x_{i,p} - d.upper \\le 0 # \\forall p, \\sum_i u_i x_{i,p} - d.lower \\ge 0 # so, two more rows per person d.constraint \u0026lt;- function(iperson, ul) { # ul = 1 for upper, 0 for lower x \u0026lt;- mat.utility.0 x[, iperson ] \u0026lt;- 1 x \u0026lt;- x * obj.utility c(as.double(x), (if (ul) c(-1,0) else c(0,-1))) } mat \u0026lt;- rbind(mat, maply(expand.grid(iperson=1:num.pers, ul=c(1,0)), d.constraint, .expand=FALSE)) dir \u0026lt;- c(dir, c(rep(\u0026#39;\u0026lt;=\u0026#39;, num.pers), rep(\u0026#39;\u0026gt;=\u0026#39;, num.pers))) rhs \u0026lt;- c(rhs, rep(0, num.pers*2)) The constraint inequalities then becomes as follows:\n\u0026gt; print(mat, digits=2) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1.00 0.000 0.000 0.000 0.00 0.000 1.00 0.00 0.000 0.00 0.00 0.000 1.00 0.0 0.000 0.000 0.0000 0.00 0 0 0.00 1.000 0.000 0.000 0.00 0.000 0.00 1.00 0.000 0.00 0.00 0.000 0.00 1.0 0.000 0.000 0.0000 0.00 0 0 0.00 0.000 1.000 0.000 0.00 0.000 0.00 0.00 1.000 0.00 0.00 0.000 0.00 0.0 1.000 0.000 0.0000 0.00 0 0 0.00 0.000 0.000 1.000 0.00 0.000 0.00 0.00 0.000 1.00 0.00 0.000 0.00 0.0 0.000 1.000 0.0000 0.00 0 0 0.00 0.000 0.000 0.000 1.00 0.000 0.00 0.00 0.000 0.00 1.00 0.000 0.00 0.0 0.000 0.000 1.0000 0.00 0 0 0.00 0.000 0.000 0.000 0.00 1.000 0.00 0.00 0.000 0.00 0.00 1.000 0.00 0.0 0.000 0.000 0.0000 1.00 0 0 1.00 1.000 0.000 0.000 0.00 0.000 1.00 1.00 0.000 0.00 0.00 0.000 1.00 1.0 0.000 0.000 0.0000 0.00 0 0 0.47 0.030 0.019 0.043 0.41 0.029 0.00 0.00 0.000 0.00 0.00 0.000 0.00 0.0 0.000 0.000 0.0000 0.00 -1 0 0.00 0.000 0.000 0.000 0.00 0.000 0.22 0.25 0.033 0.35 0.05 0.095 0.00 0.0 0.000 0.000 0.0000 0.00 -1 0 0.00 0.000 0.000 0.000 0.00 0.000 0.00 0.00 0.000 0.00 0.00 0.000 0.22 0.5 0.030 0.016 0.0096 0.23 -1 0 0.47 0.030 0.019 0.043 0.41 0.029 0.00 0.00 0.000 0.00 0.00 0.000 0.00 0.0 0.000 0.000 0.0000 0.00 0 -1 0.00 0.000 0.000 0.000 0.00 0.000 0.22 0.25 0.033 0.35 0.05 0.095 0.00 0.0 0.000 0.000 0.0000 0.00 0 -1 0.00 0.000 0.000 0.000 0.00 0.000 0.00 0.00 0.000 0.00 0.00 0.000 0.22 0.5 0.030 0.016 0.0096 0.23 0 -1 \u0026gt; dir [1] \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026gt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026quot;\u0026lt;=\u0026quot; \u0026gt; rhs [1] 1 1 1 1 1 1 1 0 0 0 0 0 0 Looking at just the last row, this constraint says that the sum of the utilities of any assigned items for person #3, minus the lower limit, must be at least 0. That is essentially the definition of the lower limit, that that constraint holds true for all three people in this problem. Similar logic applies for the upper limit.\nRunning the solver with this set of inputs gives the following:\n$solution [1] 0.000 0.000 1.000 0.000 1.000 0.000 0.000 0.000 0.000 1.000 0.000 1.000 0.000 1.000 0.000 0.000 0.000 [18] 0.000 0.498 0.433 $objval [1] 1.31 $status TM_OPTIMAL_SOLUTION_FOUND 0 Person #1 got Items 3, 5 worth 0.43 Person #2 got Items 4, 6 worth 0.44 Person #3 got Items 2 worth 0.50 The last two numbers in the solution are the values of the upper and lower bounds. Note that the objective value is only 41% higher than a random assignment, but the utilities assigned to each person are much closer. Dropping the \\(\\lambda\\) value to something closer to 0 causes the weights of the parity bounds to be less important, and the solution tends to be closer to the initial result.\nScaling this up to include constraints in pricing, farm preferences, price vs. preference meta-preferences, etc., is not conceptually difficult, but would just entail careful programming. It is left as an exercise for the well-motivated reader!\nIf you’ve made it this far, I’d definitely appreciate any feedback about this idea, corrections to my formulation or code or terminology, etc!\n(Thanks to Paul Ruben and others on OR-Exchange, who helped me figure out how to think about the parity problem, and to the authors of WP-codebox and WP LaTeX for giving me tools to put nice scrollable R code and math in this post!)\n","date":1304902861,"id":"62","link":"/2011/05/making-meat-shares-more-efficient-with-r-and-symphony/","section":"post","summary":"In my previous post, I motivated a web application that would allow small-scale sustainable meat producers to sell directly to consumers using a meat share approach, using constrained optimization …","tags":["analytics","csa","operations research","programming","R"],"title":"making meat shares more efficient with R and Symphony"},{"body":" A personal interest I have is the ethical and sustainable production of food. I’ve been a member of and helped run Community Supported Agriculture groups, and my wife and I currently purchase the majority of our meat from a group of upstate NY pastured-livestock producers who sell their products through CSAs. It’s an ala-carte business model, where I place an order on a website, and the next week I pick up the frozen products cut and packaged as if for retail.\nA related way to get meat has become fairly popular recently – the meat CSA or meat share. As the NYC Meatshare group describes it, “Looking for healthy meat raised on pasture by small local farms? It’s expensive, but by banding together to buy whole animals we can support farmers and save money.” Members of a meatshare all pitch in to buy a whole animal, which is then butchered and split among the members. Here’s how a meatshare event described the 10th of a hog each member got: “Each person will get an equal amount of bacon and sausage (about 2 lbs each), chops (center \u0026amp; butt), and will divide the other cuts up as equally as possible (including ham steak, loin, organs, etc.) If you have preferences please let me know, I will do my best to accommodate. Or, you can swap with other members at my place.”\nThese two business models put a substantial burden on either the farmer (in the first case) or the consumer (in the second case). The retail model requires the farmer, or a collective of farmers, to put together a retail-ordering web site, a butchery and inventory system, and a delivery and distribution system. The meat share model takes these burdens off the farmers, but requires the consumers to set up and organize the purchase and payment system, meet at a common location, and either take what is available or perform ad hoc swaps. In a more traditional producer-consumer relationship, the supply chain, payment, inventory, and preferences-matching process is taken care of by the comodification of the animals (all cows are the same) and the services provided by a retail grocery store.\nOne could argue that that’s the third option – Whole Foods – but it sorta defeats the purpose of non-commidified, high-quality meat, and it tends to defeat the pocketbook too. No connection with the farm, just a promise of ethical standards (probably including the pointless “organic” label), and a substantial cut by middlemen. Not really an option at all.\nSo what else could be done to build sustainable relationships between animal producers and people who value high-quality, ethically produced meat? Why not leverage technology? And not just selling via web sites, but the kind of logistics technology that allows Whole Foods (and UPS, and Walmart) to efficiently get huge varieties of goods from place to place? A group at a recent food-tech hack-a-thon had the start of this idea. They put together a quick demo of a front-end web site (“groupme.at” – clever!) that would allow consumers to choose smaller sets of cuts in such a way that the whole neatly ends up with a whole animal. By setting up a platform that can be easily connected with many small producers all over the country, the problem of every producer needing to be a webmaster is eliminated. And the system to get all of the pieces to add up to whole animals reduces risk for the farmer. It’s a great start. But by leveraging additional open-source tools and some ingenuity, I think it should be possible to do even more.\nImagine a similar web site, but instead of selecting a pre-selected package of cuts, you instead indicate your preferences and price range. As animals become available, you get an emailed notification of a delivery with a set of products that are very similar to the preferences you specified. You might love pork belly and boneless loin. Your neighbor might love cured pork belly (bacon) and chops. You might hate liver, but you’d accept some pig ears every once in a while for your dog. And your neighbor might really like the fatback to render for lard, while you’d find that useless. Everyone who might be sharing in an animal indicates their preferences, and the web site would automatically give everyone as much as possible of what they like the most. Equally important, all of the parts add up to whole animals, so the farmer is not stuck with the risk of unsold inventory.\nNow imagine that after a few months, you’ve ranked the cuts of meat from Alice’s Farm 5 stars, but the ones from Bob’s Ranch only 3 stars. And you’ve told the system that you’re willing to pay more to get more of what you really want, but you neighbor tells the web site that he’s willing to make trade-offs to spend less money. You’ve essentially added other constraints, that if balanced well, will make everyone as happy as possible. Also, notice that I mentioned both boneless loin and pork chops? They’re more-or-less the same part of the animal cut different ways, so you can’t sell them both off the same half of the same animal. Now you have exclusive constraints to add into the mix. Maybe everyone’s better off if you get the boneless loin, or maybe everyone’s better off if your neighbor gets the chops. It’s easy to imagine collecting all of this information, but how do you combine it all and optimize the outcome in a utilitarian way?\nWhy, operations research and computational optimization! Write some software that plugs everyone’s constraints into a set of equations, push a button, let the computer think for a second or two, and wham, you get a solution that balances the constraints as fairly as possible! Send the cut list to the slaughterhouse and email the product lists and bills to the customers, and you’re basically done.\nIn the past, this sort of supply-chain optimization required massive computing power and complex software design. But now, there are open-source code bases for solving this sort of problem, at least at the scale needed to balance the preferences of a few farmers and a few dozen or hundred customers at a time.\nThis is the next step in leveraging technology to make at least some aspects of the supply chain for small-scale meat operations as efficient as what Purdue does, but maintaining the high quality and personal connection to the farm that many people want now. All that’s needed are some enterprising hackers to write the code and set up a scalable, configurable web platform for preference-based meatshares.\nIn my next post, I’ll demonstrate how to write code that uses one of those open-source optimization libraries to solve a small version of this problem. If you’re interested in reading R code, stay tuned!\n","date":1304899200,"id":"63","link":"/2011/05/making-meat-shares-more-efficient/","section":"post","summary":"A personal interest I have is the ethical and sustainable production of food. I’ve been a member of and helped run Community Supported Agriculture groups, and my wife and I currently purchase the …","tags":["analytics","csa","food","operations research"],"title":"making meat shares more efficient"},{"body":"For a project I’m working on at work, I’m building a predictive model that categorizes something (I can’t tell you what) into two bins. There is a default bin that 95% of the things belong to and a bin that the business cares a lot about, containing 5% of the things. Some readers may be familiar with the use of predictive models to identify better sales leads, so that you can target the leads most likely to convert and minimize the amount of effort wasted on people who won’t purchase your product. Although my situation doesn’t have to do with sales leads, I’m going to pretend it does, as it’s a common domain.\nMy data is many thousands of “leads”, for which I’ve constructed hundreds of predictive features (mostly 1/0, a few numeric) each. I can plug this data into any number of common statistical and machine learning systems which will crunch the numbers and provide a black box that can do a pretty good job of separating more-valuable leads from less valuable leads. That’s great, but now I have to communicate what I’ve done, and how valuable it is, to an audience that struggles with relatively simple statistical concepts like correlation. What can I do?\nI’m generally interested in finding better ways to build clean, intuitive, and informative visualizations of data, especially when the visualizations can leverage intuitions and skills that everyone has. For example, almost everyone has a surprisingly good approximate number sense, the ability to quickly identify about how many items are in a largish group. For example, if shown a photo of 30 oranges and a photo of 20 oranges, you would be able to immediately say that there were more oranges in the first photo, and you would happily say that that photo had a few dozen oranges in it. This psychological skill can be used to make more effective visualizations of certain types of data. Instead of comparing two quantities by lines in a chart, or even a number in a table, it may be useful to compare visual density.\nHow can this be used to make better visualizations of prediction quality? Consider the standard ways that predictive model quality is reported. I have obfuscated the test set data from the problem I mentioned above, and placed it in a public Dropbox in Rdata format. I’ve also put together an R script to demonstrate various ways of looking at the predictions and put it in a Github gist. Follow along if you’d like.\nFirst, take a look at the data frame and some summary statistics:\n\u0026gt; head(pred.df) predicted actual actual.bin 7379 0.6020833 yes 1 5357 0.5791667 yes 1 7894 0.5791667 yes 1 5893 0.5604167 yes 1 16093 0.5541667 yes 1 2883 0.5520833 yes 1 \u0026gt; summary(pred.df) predicted actual actual.bin Min. :0.000000 no :7785 Min. :0.0000 1st Qu.:0.004167 yes: 366 1st Qu.:0.0000 Median :0.016667 Median :0.0000 Mean :0.040827 Mean :0.0449 3rd Qu.:0.041667 3rd Qu.:0.0000 Max. :0.602083 Max. :1.0000 The mode predicts about 4% of the items will be in the “yes” category, which is similar to the 4.5% that actually were. Using the very flexible ROCR package, I can quickly and easily convert this data frame into an object that can then be used to calculate any number of standard measures of predictiveness. First, I calculate the AUC value, which has a very intuitive interpretation. Consider sorting the list of items from most-predicted-to-be-“yes” to least. If the predictions are good, most of the “yes” values will be relatively high in the list. The AUC is equivalent to asking, if I randomly pick a “yes” item and a “no” item out of the list, how likely is the “yes” item to be higher on the list? If the list was randomly shuffled, it would 0.5; if it were perfectly shuffled with 20/20 hindsight, the AUC would be 1.0.\n\u0026gt; # convert to their object type (labels should be some sort of ordered type) \u0026gt; pred.rocr \u0026lt;- prediction(pred.df$predicted, pred.df$actual) \u0026gt; # Area Under the ROC Curve \u0026gt; performance(pred.rocr, \u0026#39;auc\u0026#39;)@y.values[[1]] [1] 0.8237496 In this case, it’s about .82, which is probably valuable but far from perfect. Another common way of looking at this type of predictions comes from business uses, where the goal is to identify leads (or whatever) that are likely to convert to purchases. From this point of view, the goal is to lift the leads higher in the list, so that you can focus on the top of the list and got more benefit from sales effort with less work. Two common ways of looking at lift are with a decile table, which shows how much value you get by focusing on the top 10%, 20%, etc. of the list, sorted by the predictive model, and the lift chart, which visualizes the same thing by showing how much benefit over random guessing you get by looking at more or less of the sorted list. Here they are for this data:\n# decile table dec.table \u0026lt;- ldply((1:10)/10, function(x) data.frame( decile=x, prop.yes=sum(pred.df$actual.bin[1:ceiling(nrow(pred.df)*x)])/sum(pred.df$actual.bin), lift=mean(pred.df$actual.bin[1:ceiling(nrow(pred.df)*x)])/mean(pred.df$actual.bin))) print(dec.table, digits=2) decile prop.yes lift 1 0.1 0.61 6.1 2 0.2 0.69 3.4 3 0.3 0.76 2.5 4 0.4 0.80 2.0 5 0.5 0.84 1.7 6 0.6 0.90 1.5 7 0.7 0.92 1.3 8 0.8 0.95 1.2 9 0.9 0.99 1.1 10 1.0 1.00 1.0 # Lift Curve plot(performance(pred.rocr, \u0026#39;lift\u0026#39;, \u0026#39;rpp\u0026#39;)) This graph shows, not particularly intuitively in my view, that if you focus on the top 10% of the data, you get more 5 times the bang for the buck than if you focus evenly on the whole set of items. The decile table shows the same thing – the top decile is lifted by a factor 0f 6.1, and in fact you get 61% of the “yes” items in that top 10% of the data. These are very useful numbers to know, but I think there are considerably more intuitive ways of showing how the predictive model pulls the “yes” values away from the 5% base rate.\nThese more intuitive ways are not the standard graphs used in statistics and machine learning, such as the sensitivity/specificity curve and the ROC curve. Those graphs, shown below, illustrate trade-offs between accepting false positives and false negatives. Useful, yes, but to understand them you have to think about the ways you could set a threshold and what effect that threshold would have on the nature of your predictions. That’s not particularly intuitive, and the visualization doesn’t visually contrast two things, so it’s difficult to get an intuitive understanding of what has been gained.\nI’ve put some thought and some tinkering into potentially better ways of visualizing the output of predictive models. The key, I think, is to use a visualization that builds on the scatter graph. Scatter graphs are great for less-technical audiences, because you can tell them that every individual dot is a customer (widget, whatever). They can immediately see the number of items in question, and if you can plot the points on axes that make sense to them, they can go from “that dot there represents one person with this level of X and this level of Y”, to “this set of dots represents a set of people with similar levels of X and Y”, to “this graph represents everyone, and their respective levels of X and Y.” And because of skills like the approximate number sense and the ability to quickly understand visual density, scatter graphs can give a vastly better understanding of the range of a data set than summary graphs that just plot a line and maybe some error bars.\nHere are several versions of a graph that illustrates how the predictive model smears out the set of dots from the 5% base rate, disproportionately pulling the “yes” items to the right, separating at least some of them from the much larger set of “no” items. One key change from a basic scatter graph is to jitter the Y position of each point randomly, which I think makes these graphs look a little like a PCR gel image.\nThis first approach is built around a basic scatter graph, where the X axis is the predicted likelihood of being a “yes”, and the Y axis is 0 for actual “no” and 1 for actual “yes” items. On top of that is an orange line representing the base rate of about 5%, a blue line showing the smoothed ratio between “yes” and “no” items at each level of prediction, and a thin grey line showing where the blue line ought to be. In this case, the model tends to underestimate the likelihood that some items are to be “yes” items. At 50%, half of the items should be “yes” and half should be “no”, but it’s more like 3:1.\nI like this graph as it intuitively lets people see the extent to which the predictive model is separating the categories, and how much better it does than just assuming the base rate. My second approach at this combines the “smears” with another way of visualizing lift.\nIn this graph, the smeared real data is at the bottom of the graph, and the black line represents the lift, or how much better you are at identifying “yes” items by using the predictions. It’s also an intuitive way of motivating the need to draw a boundary to focus effort. When trying to convert the points at the 25% level or above, you may be ineffective 75% of the time, but you’re also more than 10 times more efficient than you would be otherwise.\nMy final attempt worth sharing is this one, which combines the dual smear approach with the cumulative value numbers from the lift table.\nNow, in addition to being able to see the density of yes and no items for various levels of the prediction, you can see what proportion of the potential “yes” values exist to the right of each level of the prediction. For example, at a threshold of 25%, you capture 40 or 45% of the “yes” items. At a 5% threshold you capture more than 70% of the “yes” items.\nI’d love some feedback on these graphs! Do you agree with my assertion that scatter graphs are more visually intuitive and easier to motivate to non-technical audiences? Do these variations on lift charts seem clearer or more valuable than traditional alternatives to you? Have I re-invented something that should be cited?\nThe R code for these graphs is available in the Github gist. I used ggplot2, naturally, which is an essential tool for exploring the space of possible visualizations without being tied down by traditional graph structures.\nIncidentally, for people interested in building graphs that leverage people’s innate visual capabilities, I recommend Kosslyn’s book, Graph Design for the Eye and Mind.\nAlso incidentally, the question of how to communicate or visualize the potentially incredibly complex sets of rules/weights/whatever inside the categorization black box is another fascinating issue, the subject of ongoing research, and maybe something I’ll write about soon.\n","date":1303689600,"id":"64","link":"/2011/04/intuitive-visualizations-of-categorization-for-non-technical-audiences/","section":"post","summary":"For a project I’m working on at work, I’m building a predictive model that categorizes something (I can’t tell you what) into two bins. There is a default bin that 95% of the things belong to and a …","tags":["analytics","dataviz","ggplot2","graphics","predictive","R","visualization","statistics"],"title":"intuitive visualizations of categorization for non-technical audiences"},{"body":" I recently attended the INFORMS Conference on Business Analytics and Operations Research, aka “INFORMS Analytics 2011”, conference in Chicago. This deserves a little bit of an explanation. INFORMS is the professional organization for Operations Research (OR) and Management Science (MS), which are terms describing approaches to improving business efficiency by use of mathematical optimization and simulation tools. OR is perhaps best known for the technique of Linear Programming (read “Programming” as “Planning”), which is a method for optimizing a useful class of mathematical expressions under various constraints extremely efficiently. You can, for example, solve scheduling, assignment, transportation, factory layout, and similar problems with millions of variables in seconds. These techniques came out of large-scale government and especially military logistics and decision-making needs of the mid-20th century, and have now been applied extensively in many industries. Have you seen the UPS “We (heart) Logistics” ad? That’s OR.\nOR is useful, but it’s not sexy, despite UPS’ best efforts. Interest in OR programs in universities (often specialties of Industrial Engineering departments) has been down in recent years, as has been attendance at INFORMS conferences. On the other hand, if you ignore the part about “optimization” and just see OR as “improving business efficiency by use of mathematical processes,” this makes no sense at all! Hasn’t Analytics been a buzzword for the past few years? (“analytics buzzword” gets 2.4 million results on Google.) Haven’t there been bestselling business books about mathematical tools being used in all sorts of industries? (That last link is about baseball.) Hasn’t the use of statistical and mathematical techniques in business been called “sexy” by Google’s Chief Economist? How could a field and an industry that at some level seems to be the very definition of what’s cool in business and technology right now be seen as a relic of McNamara’s vision of the world?\nTo answer that rhetorical question, I think it’s worth considering the many ways that organizations can use data about their operations to improve their effectiveness. SAS has a really useful hierarchy, which it calls the Eight levels of analytics. \u0026lt;li\u0026gt;Standard Reports - pre-processed, regular summaries of historical data\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Ad Hoc Reports - the ability for analysts to ask new questions and get new answers\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Query Drilldown - the ability for non-technical users to slice and dice data to see results interactively\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Alerts - systems that detect atypical conditions and notify people\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Statistical Analysis - use of regressions and similar to find trends and correlations in historical data\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Forecasting - ability to extrapolate from historical data to estimate future business\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Predictive Analytics - advanced forecasting, using statistical and machine-learning tools and large data sets\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Optimization - balance competing goals to maximize results\u0026lt;/li\u0026gt; I like this hierarchy because it distinguishes among a bunch of different disciplines and technologies that tend to run together. For example, what’s often called “Business Intelligence” is a set of tools for doing items #1-#4. No statistics per se are involved, just the ability to provide useful summaries of data to people who need in various ways. At its most statistically advanced, BI includes tools for data visualization that are informed by research, and at its most technologically advanced, BI includes sophisticated database and data management systems to keep everything running quickly and reliably. These are not small accomplishments, and this is a substantial and useful thing to be able to do.\nBut it’s not what “data scientists” in industry do, or at least, it’s not what makes them sexy and valuable. When you apply the tools of scientific inquiry, statistical analysis, and machine learning to data, you get the abilities in levels #5-#7. Real causality can be separated from random noise. Eclectic data sources, including unstructured documents, can be processed for valuable predictive features. Models can predict movie revenue or recommend movies you want to see or any number of other fascinating things. Great stuff. Not BI.\nAnd not really OR either, unless you redefine OR. OR is definitely #8, the ability to build sophisticated mathematical models that can be used not just to predict the future, but to find a way to get to the future you want.\nSo why did I go to an INFORMS conference with the work Analytics in its title? This same conference in the past used to be called “The INFORMS Conference on OR Practice”. Why the change? This has been the topic of constant conversation recently, among the leaders of the society, as well as among the attendees of the conference. There are a number of possible answers, from jumping on a bandwagon, to trying to protect academic turf, to trying to let “data geeks” know that there’s a whole world of “advanced” analytics beyond “just” predictive modeling.\nI think all of those are right, and justifiable, despite the pejorative slant. SAS’ hierarchy does define a useful progression among useful analytic skills. INFORMS recently hired consultants to help them figure out how to place themselves, and identified a similar set of overlapping distinctions: \u0026lt;li\u0026gt;Descriptive Analytics -- Analysis and reporting of patterns in historical data\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Predictive Analytics -- Predicts future trends, finds complex relationships in data\u0026lt;/li\u0026gt; \u0026lt;li\u0026gt;Prescriptive Analytics -- Determines better procedures and strategies, balances constraints\u0026lt;/li\u0026gt; They also have been using “Advanced Analytics” for the Predictive and Prescriptive categories.\nI do like these definitions. But do I like the OR professional society trying to add Predictive Analytics to the scope of their domain, or at least of their Business-focused conference? I’m on the fence. It’s clearly valuable to link optimization to prediction, in business as well as other sorts of domains. (In fact, I have a recent Powerpoint slide that says “You can’t optimize what you can’t predict”!) And crosstalk among practitioners of these fields can be nothing but positive. I certainly have learned a lot about appropriate technologies from my membership in a variety of professional organizations.\nBut the whole scope of “analytics” is a lot of ground, and the underlying research and technology spans several very different fields. I’d be surprised if there were more than a dozen people at INFORMS with substantial expertise in text mining, for example. There almost needs to be a new business-focused advanced analytics conference, sponsored jointly by the professional societies of the machine learning, statistics, and OR fields, covering everything that businesses large and small do with data that is more mathematically sophisticated (though not necessarily more useful) than the material covered by the many business intelligence conferences and trade shows. Would that address the problem of advanced analytics better than trying to expand the definition of OR?\n","date":1302825600,"id":"65","link":"/2011/04/on-analytics-and-related-fields/","section":"post","summary":"I recently attended the INFORMS Conference on Business Analytics and Operations Research, aka “INFORMS Analytics 2011”, conference in Chicago. This deserves a little bit of an explanation. INFORMS is …","tags":["conferences","data science","job titles","operations research","statistics"],"title":"On \"Analytics\" and related fields"},{"body":" Neil Saunders has an interesting (to me) blog post up this morning, with the title “Dumped on by data scientists.” He uses the use of “data scientist” in a Chronicle of Higher Ed article to rant a little bit about the term. For Neil, it’s redundant, as the act of doing science necessarily requires data; it’s insulting, as if “scientist” wasn’t cool enough and you have to add “data”; and it’s misleading, as many people who call themselves “data scientists” are actually dealing with business data rather than scientific data.\nWithout disagreeing that there’s a terminological sprawl going on, I did want to address the use of the term, and partially disagree with Neil.\nAs someone with scientific training who uses those tools to solve business problems, I certainly struggle with a description of my role. “Data Scientist” or “Statistical Data Scientist” is actually pretty good, as it correctly indicates that I use scientific techniques (controlled experiments, sophisticated statistics) to understand our company’s data. I often describe myself as a “Statistician”, too, which gets across some of the same ideas without people having to do a double take and parse a new phrase. I also sometimes describe myself as doing “Operations Research” (aka “Management Science”, although I don’t use that term), since I use some of the tools of that field, as well as of Artificial Intelligence/Machine Learning, to optimize certain objective functions.\n“Business Intelligence” actually is not that good a term for what I do, as most of what is usually called BI is about tools for better/more relevant/faster access to data for business people to use. This is not a bad thing to be doing, at all, but it’s different from the predictive and inferential statistical methods that I use in my job.\nI don’t know what the right answer is. It might depend on the precise person and their precise role. My title, for instance, is the result of a back-and-forth with my boss, HR, and others, trying to find words that have both appropriate internal and external meanings. “Technical Lead” is a rank, indicating that I run technical projects without (formally) managing people. “Inventory Optimization and Research” covers a variety of areas. “Inventory” here means “sellable units”, like boxes on a shelf, or in this case, like scheduled airline flights. Probably baffling for an external audience without an explanation, but extremely clear inside the company. “Optimization” means what it sounds like, both in a technical and a non-technical sense, and for both internal and external audiences. “Research” indicates a focus on the development of long-term and cutting-edge systems. “Data Scientist” didn’t end up in there, but it could have.\nFor people using Big Data tools and scientific methods to study topics inside academia, the right answer seems to me to put the field of study first. You’re not a “Data Scientist”, you’re an astrophysicist, or a bioinformatician, or a neuroscientist, with a specialization in statistical methods. If you’re a generalist inside the academy, you’re probably a statistician. Perhaps “Data Scientist” should be restricted to people applying scientific tools and techniques to problems of non-academic interest? That might work, as long as it included people who do things like apply predictive analytic tools to hospital admissions data.\n","date":1297555200,"id":"66","link":"/2011/02/data-scientist-and-other-titles/","section":"post","summary":"Neil Saunders has an interesting (to me) blog post up this morning, with the title “Dumped on by data scientists.” He uses the use of “data scientist” in a Chronicle of Higher Ed article to rant a …","tags":["data science","job titles","statistics"],"title":"\"Data Scientist\" and other titles"},{"body":" I was recently given the opportunity to re-present my ggplot2 talk, which I originally gave to the NYC R Meetup, to the DC R Meetup group. The Meetup was held co-located with the Predictive Analytics World conference in Alexandria, VA. (More on my thoughts on PAW below…) Contentwise, I made only small changes, changing a bit of patter and adding more examples at the end. I still love ggplot, with some frustration at the way it is typically introduced. Some of the audience had no R experience at all, while others were experts. One person, a grad student at U. of Maryland, had had very similar difficulty as I had when originally learning ggplot2, and his enthusiastic nods during my presentation were very validating! For reference, the Meetup page is here, and I stuck the current version of the slides in a public Dropbox, located here.\nAnd a few thoughts about PAW. The conference was well-run (although I have my gripes with the hotel and its location!) and there were an interesting and eclectic lineup of speakers, from a variety of industries. Compared to academic conferences I’ve attended, I missed having all the grad students around. At PAW, I felt rather young, which had not been true at academic conferences in quite a long time! The content of the conference focused on people using predictive methods (statistics, data mining, machine learning) at the individual-customer level, for marketing or retainment or other purposes. That’s not my primary interest right now – my work is focused at a slightly higher operations-research-y level, trying to make sure that customers in the aggregate have good options. But I enjoyed learning about what other people are doing using somewhat similar methods. Next year, though, I think I’ll try to go to a different conference, perhaps UseR! in the UK, or INFORMS’ applied conference…\n","date":1287878400,"id":"67","link":"/2010/10/how-to-speak-ggplot2-like-a-native-and-predictive-analytics-world/","section":"post","summary":"I was recently given the opportunity to re-present my ggplot2 talk, which I originally gave to the NYC R Meetup, to the DC R Meetup group. The Meetup was held co-located with the Predictive Analytics …","tags":["ggplot2","meetup","R"],"title":"how to speak ggplot2 like a native, and Predictive Analytics World"},{"body":" The Meetup phenomenon, which is now substantial and longstanding enough to be more of a cultural change than a flash in the pan, continues to impress me. Even more so than tools like LinkedIn, Meetups have changed the nature of professional networking, making it more informal, diverse, and decentralized. Last night, statistics consultant (and cheap eats guru) Jared Lander and I presented a talk on a statistical technique tangentially related to my professional work (more closely associated with Jared’s). The origin of this presentation is worth noting. On Meetup’s web site, members of a group can suggest topics for meetings. Before even attending a single NYC Predictive Analytics event, I posted several topics that I thought might be interesting for the group. A bit later, the organizers (Bruno and Alex) contacted me to see if I’d be willing to present on prediction with Multilevel models. I said that I would, but only if I could co-present with someone who actually knew something about the topic a complementary set of skills and experiences. Knowing Jared from the NYC R Meetup group, and knowing that he learned about multilevel models from the professor who wrote the best book on the topic, and knowing that he’s pretty good in front of an audience, I suggested we collaborate.\nDespite requiring a lot of work, and a lot of learning of details on my part, we managed to throw together a pretty decent talk. (As of this morning, there’s four ratings of the event on Meetup, and we got 5/5 stars! Yay us! Not statistically conclusive, though…) We used as an example topic for data analysis the difficult and critically important problem of predicting reviews of pizza restaurants in downtown NYC. Jared is actually an expert on this topic, having written his Masters thesis on ratings from Menupages.com. For the talk, Jared would present a few slides, then I’d present a few. In a few cases we’d both try to explain topics from slightly different points of view. I’d repeatedly try to use the keyboard instead of the remote-control gadget to control Powerpoint, causing the computer to melt down into a pile of slag and refuse to change the slide. Jared would send me withering glares when I started to move towards the keyboard. It ended up OK, though, we got through everything, and even answered about half of the (excellent) questions! Oh, and shout-out to the AV guy at AOL HQ. I don’t know how they pay his salary, but he rocked.\nJared has posted the slides from the talk here (ppt), and I’ve put the data we made up (for pedagogical purposes) and the code we used to analyze it and generate graphs for the talk here on Github. Alex video-recorded the presentation, and I’ll update this sentence to link to the video once it’s posted somewhere. Hope folks find it valuable!\n","date":1287100800,"id":"68","link":"/2010/10/prediction-with-multilevel-regression-models-and-pizza/","section":"post","summary":"The Meetup phenomenon, which is now substantial and longstanding enough to be more of a cultural change than a flash in the pan, continues to impress me. Even more so than tools like LinkedIn, Meetups …","tags":["meetup","statistics","R"],"title":"Prediction with Multilevel Regression Models, and Pizza"},{"body":"As more and more people get smartphones that can play MP3s or streamed music, like the iPhone or Android phone like the upcoming HTC Evo 4G (I’m gettin’ one!), fewer and fewer people are buying standalone MP3 players. Why have two gadgets when you can have just one? But I think there are good reasons to do so, but I don’t think the right combination of products are currently on the market. Here’s my thinking. First of all, you want to be able to make or answer calls from the gadget that’s playing your music. It’ll automatically turn off your music when the phone rings, and you can hear the caller through your headphones. This works fine in current smartphones. Good sound quality, and you can use your own headphones. But you still need a cord between your headphones and your phone. And that cord gets caught in things, and it gets in the way when you want to actually use your phone. Clunky.\nSolution: Bluetooth. There are two current ways to use bluetooth gadgets with a smartphone. First of all, you can get a stereo bluetooth headset. Here’s one that looks like the common one-ear bluetooth headsets for phone use, plus a short cord to a separate earphone for your other ear. There are a variety of options, but they all have the same basic idea. Integrated earphones with microphone, and an A2DP connection between your phone and the gadget. A2DP streams medium-quality stereo sound in real-time in one direction, plus you can do simple things like tell your phone to pick up a call and stop the music by pushing a button on your headset. Sounds great, but there are problems. You have to use the earphones provided, which means your studio-quality Shure or noise-cancelling Bose headphones are useless. Second, the audio quality of A2DP is not that great.\nA half-solution is a stereo bluetooth gadget that hangs from your neck or clips on your shirt, with a jack so you can use your own headphones. I have one of these, and it sorta works. You can push a button to answer the phone, and it plays the music. But with the better earphones, it’s easy to hear the compression issues with A2DP, and there’s not much ability to control the music without digging my phone out of my pocket. I mostly use it as a bluetooth telephone headset, and rarely to listen to music. Plus, the processing needed to decode and recode the audio is significant, so my phone is not very useful for much else when it’s streaming sound. So the product fails at its intended use.\nInstead, I use a dedicated MP3 player. I’ve got a Cowon iAudio 7, and in many ways it’s great. Amazing sound quality, plays OGG files, FM radio, very nice OLED screen. But it doesn’t talk to my phone. Why not?\nHere’s what I want. I want my MP3 player and my phone to be two separate interfaces to the same audio sources. The MP3 files should live on my phone, synced to the cloud or my computer, and of course the phone should pull down Internet content. But my phone should push still-compressed versions of the audio content up to my MP3 player, which can take responsibility for decoding it and sending out to my earphones. I want a small eInk or OLED screen, a microphone, and a few simple buttons on the MP3 player – just enough to change the volume, skip to the next track, and see what’s playing. But I want the rest of the controls (plus duplicates of the ones on the MP3 player) to live on my phone, with a big gorgeous color screen, and lots of room for menus, fancy controls, and all the rest. If I play video on my phone, I want the audio to stream, perfectly-sync’ed, with no loss of quality, to my ears. If the phone rings, I want the ability to use either buttons on the MP3 player, or on my phone, to stop the music and pick up. I want the MP3 player to be a well-integrated accessory to my phone, not an afterthought, and I want each piece in the system (smartphone, MP3 player with microphone, and headphones) to do the things they do the best. As far as I know, the current state-of-the-art is nowhere close to this, but there are no good reasons why not.\nDrop me a line when your product that does this is on the market. I’ll buy one.\n","date":1269648e3,"id":"69","link":"/2010/03/smartphones-mp3-players-and-bluetooth-the-division-of-labor/","section":"post","summary":"As more and more people get smartphones that can play MP3s or streamed music, like the iPhone or Android phone like the upcoming HTC Evo 4G (I’m gettin’ one!), fewer and fewer people are buying …","tags":["bluetooth","gadgets","mp3","phones","smartphones"],"title":"Smartphones, MP3 players, and Bluetooth: the division of labor"},{"body":"A few months back I gave a presentation to the NYC R Meetup. (R is a statistical programming language. If this means nothing to you, feel free to stop reading now.) The presentation was on ggplot2, a popular package for generating graphs of data and statistics. In the talk (which you can see here, including both my slides and my patter!) I presented both the really great things about ggplot2 and some of its downsides. In this blog post, I wanted to expand a bit on my thinking on ggplot, the Grammar of Graphics, and how peoples’ conceptual representations of graphs, data, ggplot, and R all interact. ggplot is both incredibly elegant and unfortunately difficult to learn to use well, I think as a consequence of the variety of representations. The ggplot package, written by the overachieving and remarkable Hadley Wickham, is based on earlier more theoretical work by Leland Wilkinson. Wilkinson abstracted the process of putting data onto an image, and created a Grammar of Graphics, which describes how the data maps to the parts of a graph, rather than describing the final graph itself. For example, here’s how to create a pie chart, clipped from Wilkinson’s book:\nDon’t worry about the details, but briefly, a pie chart is just a stacked bar graph (summary.proportion) plotted in polar coordinates (polar.theta). If you took the time to learn this grammar, you would realize that the hierarchical structure of a graph on a page (elements have positions and labels and visual properties like color, each of which have their own abstract structure) maps cleanly to the hierarchical structure of the grammar, and that variables in the grammar map cleanly to the linear structure of the data. As a user of this system, you would be able to see all three key representations at once: the data, the grammatical mapping from data to graph, and the graph itself.\nNow consider ggplot, the implementation of the Grammar of Graphics in the R programming language. Does ggplot maintain three visible representations, all straightforwardly mappable to each other? Sadly, it does not. Instead, users of ggplot must map among four representations: the data (a standard data.frame object), the R syntax for ggplot2 (which has some quirks), an underlying ggplot object (similar to the Grammar of Graphics, but vastly more complex and impossible to examine directly), and the generated graph.\nConsider the simple pie graph, below.\nThis chart is generated in ggplot2 by the following R code:\n\u0026gt; zz \u0026lt;- data.frame(cat=c(\u0026quot;a\u0026quot;, \u0026quot;b\u0026quot;), val=c(5,3)) \u0026gt; zz cat val 1 a 5 2 b 3 \u0026gt; pp \u0026lt;- ggplot(zz, aes(x=\u0026quot;\u0026quot;, y=val, fill=cat)) + geom_bar(width=1) + coord_polar(\u0026quot;y\u0026quot;) \u0026gt; print(pp) The print() function is optional within an R interpreter session, but I include because it illustrates a point that’s not initially obvious to many users. Unlike the built-in R plotting tools, the ggplot() function and its associated functions don’t plot anything on the screen, they just construct an object of type “ggplot”. Almost all of the actual work of mapping the data to stuff on your screen occurs when you print that object, using print() or ggsave().\nSo what does that object look like? If you type str(pp), you’ll get an answer, but it’s about a hundred lines of undecipherable hierarchical object and list structure, not intended to be examined by mere mortals. But there’s something critically important about that structure – like the original Grammar of Graphics, and unlike the R syntax above, it’s hierarchically structured.\nIn the R syntax, you create a base ggplot structure with the ggplot() call, then you abuse the “+” operator to make changes to that structure. The geom_bar() function adds a layer to the ggplot() object, where a layer is just what it sounds like, a set of information about one of potentially many overlaid layers of content that will be put on the graph. So you construct a ggplot object by first initializing everything about the basic plot, then tack on layers with +, right? Actually no, because the coord_polar() call doesn’t create or modify a layer at all, it modifies the base object! Even if you’ve acquired the nonobvious intuition that ggplot objects are hierarchical and are created by concatenating layers, you now have to break the analogy again to fully understand what + is doing!\nThere is a way to partially see the structure directly, but it’s not well thought-out from the point of view of someone trying to learn how to use the package. The summary() method on ggplot objects tells you about things you didn’t specify (faceting?), it’s incomplete, and it doesn’t map well to the R syntax. If something in your plot isn’t working the way you want it to, summary() won’t help you.\n\u0026gt; summary(pp) mapping: x = , y = val, fill = cat faceting: facet_grid(. ~ ., FALSE) ----------------------------------- geom_bar: stat_bin: width = 1 position_stack: (width = NULL, height = NULL) Another shortcut that leads to conceptual problems by ggplot beginners is the use of qplot(). The qplot() function is a wrapper around ggplot(). Unlike ggplot(), you can give qplot() data that is not in the form of a data.frame, and the syntax is somewhat different. There’s nothing wrong with some syntactic sugar to make life easier, but in this case, learning ggplot by starting with qplot is like trying to learn a foreign language by starting with contractions and slang. You may be able to say a few essential things on your vacation, but you won’t be able to creatively construct new sentences as new situations arise. The brilliance of the Grammar of Graphics is exactly that it’s a grammar – you can construct new graphs and new types of graphs as new situations arise! But tutorials that start with qplot, with the ggplot book an unfortunate (but in other ways excellent) example, send their learners down a linguistic garden path. To fully use the power of the system requires unlearning the conceptual structures that map the slang to charts on a screen, and starting over with learning the new, more powerful ggplot() grammar and hierarchical representations.\nI’d like to conclude this overlong rant with two notes. First, just today a new graphics package for R was introduced. jjplot uses many of the ideas of the Grammar of Graphics and ggplot2, but seems to avoid at least a few of the conceptual problems. The + operator is not overloaded in conceptually confusing ways, and there is no distracting qplot function to mislead new users. Additionally, a quick look at the source code finds it much, much simpler than ggplot2’s source, which will likely lead to a more active base of contributors. I look forward to trying jjplot and watching its continuing development, and hope the authors learn from both the remarkable successes and frustrating failures of ggplot. Second, I use ggplot extensively in my work. It’s simply the best available tool for quickly generating elegant graphs of data in R, especially if that generation needs to happen automatically in code. Hadley Wickham deserves extensive praise for the amount of effort he has put into developing and popularizing the Grammar of Graphics. If you want to be maximally effective when visualizing data in R, take the time to learn ggplot2, but do so while keeping in mind that the learning process will be easiest if you skip qplot and other shortcuts, think hierarchically, and prepare for some frustration. Fortunately, the support communities on the ggplot mailing list and Stack Overflow are extremely helpful, as is Hadley himself.\n","date":126792e4,"id":"70","link":"/2010/03/ggplot-and-concepts-whats-right-and-whats-wrong/","section":"post","summary":"A few months back I gave a presentation to the NYC R Meetup. (R is a statistical programming language. If this means nothing to you, feel free to stop reading now.) The presentation was on ggplot2, a …","tags":["ggplot2","graphics","programming","R"],"title":"ggplot and concepts -- what's right, and what's wrong"},{"body":" Hah, it rhymes! The fall haul (hah!) from the CSA inevitably means two things, root vegetables and ungodly numbers of pears. I love root vegetables, but tend to find pears to be pale imitations of apples. But when poached in red wine, or cooked with butter and sugar, pears can have some redeeming value. Recently, for the cooking club, I teamed up to make a dessert with the theme “Fall Harvest.” We were inspired by a recent recipe from the late, lamented Gourmet magazine, Beet and Pear Napoleons with Ginger Juice Vinaigrette. Roasted beets layered with pears, with flavors of ginger, citrus, tarragon, and poppy. In fact, we were so inspired by the dish, that we made it too, and started the dinner with an amuse of the original recipe:\nFor the dessert itself, we took these flavors and turned them on their heads. The formerly raw pear circles were now lightly carmelized with sugar and butter. The beet, instead of being roasted, was boiled, pureed, and folded into a chocolate cake batter. The ginger juice was mixed into a buttercream instead of a vinagrette. The texture and visual impact of the poppy seeds was replaced with cocoa nibs. The result was an almost perfectly balanced mixture of fruit and butter flavors, sweet and tart, creamy and crunchy. (If I were to make it again, I’d actually add more pear!) We were very happy with the result:\n(recipe outline here)\n","date":1256256e3,"id":"71","link":"/2009/10/savory-and-sweet-pears-and-beets/","section":"post","summary":"Hah, it rhymes! The fall haul (hah!) from the CSA inevitably means two things, root vegetables and ungodly numbers of pears. I love root vegetables, but tend to find pears to be pale imitations of …","tags":["cooking club","csa","dessert","food","recipes"],"title":"Savory and sweet, pears and beets"},{"body":" The problem of how to monetize online publishing, particularly news publishing, is neither new nor all that surprising. But the ongoing lack of a solution is steadily eating into news organizations across the country. Yesterday, the Times announced it was going to buy out or lay off 8% of its newsroom staff, despite being the best national newspaper in the country and probably the one making the best use of Internet technologies. (Their interactive graphics are some of the best around.) How can newspapers make money on the web? Ad revenue is inadequate, and people won’t generally pay for content. This post from a journalism blog at Harvard discusses why micropayments will never work:\nApple can charge for music because it controls access to the songs from all the major record labels. Phone companies and cable companies can charge usurious rates for text messaging and Internet because they have little or no real competition. How does any of that apply to newspapers? … Newspapers have spent the past 100 years or so with a stranglehold on both the tools of mass publishing and the means of distribution, and much of what has happened to them over the past decade is a result of them losing both of those things. The unfortunate reality is that even the best micropayment system is not going to recreate that system of artificial scarcity and control…\nBut I think there’s a way that might work, a way that leverages human psychology. People like to feel like they’re in control, and they like to feel like they have a voice in the system. Micropayment systems that require you to pay 10 cents to read an article, based on a headline or a link, or subscription systems that take your money and give you something you can get elsewhere for free, just make you resentful. So instead, design the system so that you associate feeling good about what you have just read with giving money to the people who produced the content. Here’s how it might work.\nIf I decide I want to read content from a consortium of providers (say, anything owned by The New York Times Company, or Time-Warner, or Seed Media Group, or a group of publishers that set up their own consortium), I set up an account, pay my $50/year, and get access. If I like a piece of content (article, podcast, interactive graphic, whatever), I click the “Tip the Author(s)” button, and a chunk of my $50, maybe 10 cents, gets redirected to the actual people creating the content I actually like (not just start to read). If I don’t use up my $50 for the year, the balance just gets split internally by the consortium. This way, readers have a feeling of control and an association of paying with pleasure, providers get cash, and the best providers get the most cash.\nInformation management for this would be straightforward, and it would (I think) work. People like to tip for good service. Let them tip for informative, well-reported news.\n","date":1255996800,"id":"72","link":"/2009/10/online-publishing-micropayments-and-warm-fuzzy-feelings/","section":"post","summary":"The problem of how to monetize online publishing, particularly news publishing, is neither new nor all that surprising. But the ongoing lack of a solution is steadily eating into news organizations …","tags":["internet","journalism","microeconomics","money"],"title":"Online publishing, micropayments, and warm fuzzy feelings"},{"body":"Publications Public Presentations Panel: Staff+ IC Data Careers. (2022) Locally Optimistic Meetup. With Paige Berry and Terry Joyce.\nA/B Testing and Decision-Making. (2022) Quant UX Conf 2022.\nThe Software Architecture of WayUp's Job Recommender System. (2018) DataEngConf NYC 2018.\nCold-Start Recommendations to Users With Rich Profiles. (2018) RecSys NYC Meetup.\nCollecting and Making Sense of Diverse Data at WayUp. (2017) DataEngConf 2017, New York City.\nAPIs and DSLs for Building and Integrating Many Models. (2017) PAPIs 2017, Boston.\nForecasting Repeated Accumulating Processes with Semiparametric Regression Models and Bayesian Updates. (2017-2018) NY Open Statistical Programming Meetup; International Society for Business and Industrial Statistics Conference 2017, Westchester, NY; Conference on Statistical Practice 2018, Portland, OR.\nBig Data, Public Policy, Higher Ed, \u0026amp; Industry: Statistics, Challenges, \u0026amp; Opportunities. (2016) WSS Big Data in the Public Sector Mini-Conference.\nShiny Apps for Repeated Modeling Workflows. (2016) Shiny Developers Conference.\nPredicting Student Success at Scale — APIs and DSLs for Building and Integrating Many Models. (2015) EARL Conference.\ndplyr and Databases. (2015) Data Wranglers DC Meetup.\nSoftware Architecture \u0026amp; Predictive Models in R. (2015) NYC R Conference.\nJulia: applying language design lessons to technical computing. (2014) Polyglot Programming DC Meetup.\nHow to Put Your Meetup on the Map (Literally). (2013-2014) INFORMS MD, Statistical Programming DC, New York Open Statistical Programming. Predictive Analytics World – Government (invited). With Alan Briggs.\nWhy a Data Community is like a Music Scene. (2013) Strata NYC Ignite talk.\nPanel: Creating and Sustaining a Data Community. (2013) DataGotham. With Matt Turck and Noah Hidalgo.\nGrowing Pains: Talking About Data Scientists. (2012) DataGotham.\nAnnotating Enterprise Data from an R Server. (2012) DC UseR Group.\nWhat is “Data Science” Anyway? (2011) Data Science DC Meetup.\nAn Introduction to Multilevel Regression Modeling for Prediction. (2010) NYC Predictive Analytics Meetup. With Jared Lander\nHow to Speak ggplot2 Like a Native. (2010) DC UseR Group.\nDemystifying error messages and debugging in R — Advanced topics. (2010) New York R Statistical Programming Meetup.\nThe R Rosetta Stone — Matlab. (2010) New York R Statistical Programming Meetup. With Marck Vaisman.\nIntroduction to the Grammar of Graphics with ggplot2 in R. (2009) New York R Statistical Programming Meetup.\nNon-technical Articles Harris, H. D., Murphy, S. P., \u0026amp; Vaisman, M. (2013). Analyzing the Analyzers: An Introspective Survey of Data Scientists and Their Work. O’Reilly Media.\nPapers in Refereed Journals Magnuson, J. S., Mirman, D., Luthra, S., Strauss, T., \u0026amp; Harris., H. D. (2018). Interaction in spoken word recognition models: Feedback helps. Frontiers in Psychology.\nHarris, H. D., Murphy, G. L., \u0026amp; Rehder, B. (2008). Prior knowledge and exemplar frequency. Memory \u0026amp; Cognition, 36, 1335-1350.\nHoffman, A. B., Harris, H. D., \u0026amp; Murphy, G. L. (2008). Prior knowledge enhances the category dimensionality effect. Memory \u0026amp; Cognition, 36, 256-270.\nStrauss, T., Harris, H. D., \u0026amp; Magnuson, J. S. (2007). jTRACE : A reimplementation and extension of the TRACE model of speech perception and spoken word recognition. Behavior Research Methods, 39, 13-30.\nDell, G. S., Lawler, E., Harris, H. D., \u0026amp; Gordon, J. K. (2003). Models of errors of omission in aphasic naming. Cognitive Neuropsychology, 21, 125-145.\nDell, G. S., Harris, H. D., \u0026amp; Guest, D. J. (2001). Erreurs de production, contraintes phonotactiques, et expérience récente (Speech errors, phonotactic constraints, and recent experience). Psychologie Français, 46, 55-64.\nPapers Under Submission Harris, E. S., Harris, H. D., \u0026amp; Malkovsky, M. (submitted). Blood Type Distribution in Autoimmune Diseases: An Anonymous, Large-Scale, Self-Report Pilot Study.\nBook Chapters Magnuson, J. S., Mirman, D., \u0026amp; Harris, H. D. (2012). Computational models of spoken word recognition. M. Spivey, K. McRae \u0026amp; M. Joanisse (Eds.), Cambridge Handbook of Psycholinguistics, 76-103.\nHarris, H. D. \u0026amp; Rehder, B. (2011). Knowledge and resonance in models of category learning and categorization. E. Pothos \u0026amp; A. Wills (Eds.), Formal Approaches to Categorization, 274-298.\nPapers in Refereed Conference Proceedings Harris, H. D. (2018). An Architecture and Domain Specific Language Framework for Repeated Domain-Specific Predictive Modeling. In The Proceedings of Machine Learning Research, 4th International Conference on Predictive Applications and APIs.\nHarris, H. D. (2008). Categorizing Fragments of Exemplars: Experimental and Computational Results. In B. C. Love, K. McRae, \u0026amp; V. M. Sloutsky (Eds.), The Proceedings of the 30th Annual Meeting of the Cognitive Science Society (pp. 409-414). Austin, TX: Cognitive Science Society.\nHarris, H. D. \u0026amp; Minda, J. P. (2006). An attention-based model of learning a function and a category in parallel. In The Proceedings of the 28th Annual Meeting of the Cognitive Science Society.\nHarris, H. D. \u0026amp; Rehder, B. (2006). Modeling category learning with exemplars and prior knowledge. In The Proceedings of the 28th Annual Meeting of the Cognitive Science Society.\nMagnuson, J. S., Strauss, T., \u0026amp; Harris, H. D. (2005). Interaction in spoken word recognition models: Feedback helps. In The Proceedings of the 27th Annual Meeting of the Cognitive Science Society.\nStrauss, T., Magnuson, J. S., Pelosof, R., \u0026amp; Harris, H. D. (2005). jTRACE: A reimplementation and extension of TRACE for research and education. In The Proceedings of the 27th Annual Meeting of the Cognitive Science Society.\nHarris, H. D., \u0026amp; Minda, J. P. (2005). Function learning with an ensemble of linear experts and off-the-shelf category-learning models. In The Proceedings of the 27th Annual Meeting of the Cognitive Science Society.\nReichler, J. A., Harris, H. D, \u0026amp; Savchenko, M. A. (2004). Online parallel boosting. In The Proceedings of AAAI-2004.\nHarris, H. D., \u0026amp; Kiefer, S. M (2004). A survey of instructors of Introductory Artificial Intelligence. In The Proceedings of FLAIRS-2004.\nHarris, H. D. (2002). Holographic reduced representations for oscillator recall: A model of phonological production. In The Proceedings of the 24th Annual Meeting of the Cognitive Science Society.\nHarris, H. D. (2002). Evidence that Incremental Delta-Bar-Delta is an attribute-efficient linear learner. In Machine Learning: ECML 2002, 135-147, Springer.\nHarris, H. D. \u0026amp; Reichler, J. A. (2001). Learning in the cerebellum with sparse conjunctions and linear separator algorithms. In The Proceedings of the International Joint Conference on Neural Networks 2001.\nWorkshop Papers, Technical Reports, and Posters Harris, H. D., Hoffman, A. B., \u0026amp; Murphy, G. L. (2007). Prior Knowledge and Non-minimal Category Learning: Experimental and Modeling Results. Poster presented at The 48th Annual Meeting of the Psychonomic Society.\nMagnuson, J. S., Strauss, T., \u0026amp; Harris, H. D. (2005) On the role of interaction in models of spoken word recognition: Feedback helps. Poster presented at The 18th Annual CUNY Sentence Processing Conference.\nHarris, H. D., \u0026amp; Magnuson, J. S. (2004) Proper names, common nouns, and category learning. Poster presented at The 45th Annual Meeting of the Psychonomic Society.\nHarris, H. D., \u0026amp; Magnuson, J. S. (2004) A cross-disiplinary look at statistics and grounding in human lexical learning. In The Working Notes of the AAAI-2004 Spring Symposium on Interdisciplinary Approaches to Language Learning.\nHarris, H. D., \u0026amp; Kiefer, S. M. (2003). A survey of instructors of Introductory Artificial Intelligence. UIUC Department of Computer Science Technical Report UIUCDCS-R-2002-2313.\nHarris, H. D. (2003). New Algorithms for Attribute-Efficient On-Line Linear Learning. UIUC Department of Computer Science Technical Report UIUCDCS-R-2003-2352.\nReichler, J. A., \u0026amp; Harris, H. D. (2001). Parallel Online Continuous Arcing and a new framework for wrapping parallel ensembles. In The Working Notes of the IJCAI-01 Workshop on Wrappers for Performance Enhancement in KDD.\n","date":0,"id":"73","link":"/publications/","section":"","summary":"Publications Public Presentations Panel: Staff+ IC Data Careers. (2022) Locally Optimistic Meetup. With Paige Berry and Terry Joyce.\nA/B Testing and Decision-Making. (2022) Quant UX Conf 2022.\nThe …","tags":[],"title":""},{"body":" NYTimes-ReSection Privacy Policy This is a placeholder privacy policy page for the NYTimes-ReSection Chrome extension.\nReplace the contents of this file with your full privacy policy HTML. The URL of this page will remain the same, so you can safely link to it from the Chrome Web Store listing and from within the extension.\n","date":0,"id":"74","link":"/nytimes-resection-privacy/","section":"","summary":"NYTimes-ReSection Privacy Policy This is a placeholder privacy policy page for the NYTimes-ReSection Chrome extension.\nReplace the contents of this file with your full privacy policy HTML. The URL of …","tags":[],"title":"NYTimes-ReSection Privacy Policy"}]