Professional on Harlan D. Harris

A Big Win from a Small Search UI Change and a Predictive Model

harlan@harris.name (Harlan Harris) — Thu, 09 Jul 2026 00:00:00 +0000

One of my biggest commercial wins was a seemingly small UI tweak that led to a roughly 2% marketplace revenue bump. Teachers Pay Teachers (TPT, think "Etsy for Teachers" -- educational resources created by and sold to other educators) was a mature marketplace, where revenue wins were typically under 0.5%. How we got this win I think is instructive, covering the value of a search scientist on a search team, and a holistic view of search UI/UX and algorithms.

Design Decisions for a New Search Results Page

harlan@harris.name (Harlan Harris) — Fri, 08 May 2026 00:00:00 +0000

One of my specialties as a freelance consultant is discovery systems for e-commerce -- web pages and backend systems for helping users search for and find what they want or need. For the last year and a half, I've been working as a consultant for DesignShop, a company that sells materials samples for home renovations. Want to find a countertop, cabinet, paint, and tile combo for your kitchen reno that looks great, with samples shipped overnight? Check 'em out.

You Don't Need a Search Engineer; You Need a Search Scientist

harlan@harris.name (Harlan Harris) — Mon, 13 Apr 2026 00:00:00 +0000

Say you're an Engineering Manager or a Product Manager, responsible for a business-critical search system on a website. Maybe e-commerce, maybe a marketplace, maybe something else. Search is sort of working, it doesn't crash, and it's fast, but there's a feeling that "search isn't great", and you've been tasked with finding someone to help. Your first instinct, or that of your VP, might be some version of: “we need to hire a search engineer.”

Nine Things I Learned Building a Scrabble App for My Friends

harlan@harris.name (Harlan Harris) — Sun, 29 Mar 2026 00:00:00 +0000

I grew up playing Scrabble -- my grandma was cuthroat. I've played occasionally through the years, and more recently I've been playing Scrabble Go, the official, licensed Scrabble app, with friends. It's fine, but it's full of ads (unless you pay them), it has annoying side games (gems? why am I earning gems?), and I wished it had features like word definitions and better game history. So, starting last year, I built my own web app. I've had half a dozen friends and family members playing with me, each other, and a robot for the last few months, and it's been a lot of fun. I thought I'd share a few things I've learned along the way.

IZE meets AI -- semantic search, smarter labels, and agentic orientation

harlan@harris.name (Harlan Harris) — Wed, 18 Mar 2026 00:00:00 +0000

This is the last post in the IZE series. In the previous installment, I looked at two ways to generalize the IZE algorithm itself: preferring consistent facets and searching for trees with better goodness scores. Here I want to ask a different question: what does the AI revolution of the last few years actually change about what could be built here?

I see three distinct opportunities, at different layers of the system.

GeneralIZE -- How else could IZE's hierarchies be generated?

harlan@harris.name (Harlan Harris) — Tue, 17 Mar 2026 00:00:00 +0000

In previous posts, I showed off an interactive demo of the IZE algorithm, and discussed how the algorithm worked. Now, it's worth considering some ways we could generalIZE the algorithm. 🤦‍♂️ Perhaps variations on the algorithm might yield hierarchies that are even better at showcasing the contents of the texts?

As discussed previously, there's pretty good experimental evidence that in order to be useful and intuitive, search results clustering needs to be based on just one or two words (or facet values, or tags). That is, it's too confusing when the algorithm finds clusters based on overall family resemblance, then tries to name them post-hoc. It's better to do the sort of single-word splitting that the IZE algorithm uses. But, could we do so in a different way or a different order, so that the results feel more like underlying clusters are being identified?

Could We Build IZE Again?

harlan@harris.name (Harlan Harris) — Mon, 09 Mar 2026 00:00:00 +0000

In the previous installment of this series, I looked at what came after IZE -- faceted search, clustering algorithms, and the various ways web search, personal information management, and e-commerce tried to solve similar problems to what IZE was attacking. None of them ended up doing what IZE did. The question I want to take up here is: could we build something like IZE today? I think the answer is yes, for at least one domain. IZE's hierarchical search algorithm is worth revisiting for e-commerce operators with medium-to-large catalogs who want to help new customers orient themselves. Modern catalog metadata makes it easier to implement than ever.

What Came After IZE? Three Domains, Three Answers

harlan@harris.name (Harlan Harris) — Wed, 04 Mar 2026 00:00:00 +0000

In the previous post in this series, I discussed the technical details of IZE and its reception. Here I want to look at what came after — and where IZE-like ideas might still have potential.

The short version: IZE was forgotten, but the ideas it embodied — hierarchical clustering, single-word splits, dynamic navigation — were re-invented independently in several different domains. Each domain found a different answer, for reasons that are worth understanding.

How IZE Really Worked - Algorithm, Patent, Limits, and Esther Dyson

harlan@harris.name (Harlan Harris) — Sat, 28 Feb 2026 00:00:00 +0000

In the first post in this series, I introduced IZE -- a DOS-era personal information manager with a novel approach to search and navigation. Here I want to go deeper into how it actually worked, what its limits were, and how it was received at the time.

The algorithm

The core of IZE was patented by Paul Kleinberger (US5062074A, "Information retrieval system and method"). The basic algorithm is easy enough to describe:

IZE - Revisiting a hierarchical search technology from the PC era

harlan@harris.name (Harlan Harris) — Fri, 27 Feb 2026 00:00:00 +0000

Sometimes revisiting old technology is the best way to understand how we got where we are -- and to see what alternative paths might have looked like. This is the first in a series of posts about IZE, a DOS-era personal information manager that I think has some interesting lessons for modern search and discovery.

I'll admit my interest in IZE isn't purely academic. My father was a co-founder of the company that published it, and a cousin of his was the inventor of the underlying technology, which the company acquired. So I grew up with IZE around the house, and owned the polo shirt. That said, I think it's genuinely interesting on its own merits, and I'll try to make the case for that here.

11 Algolia A/B Testing Gotchas, Tips, and Lessons!!

harlan@harris.name (Harlan Harris) — Mon, 09 Feb 2026 00:00:00 +0000

I've supported A/B testing of Algolia search systems at three companies now, and have dived deep into A/B testing generally as well as specifically for search. The Algolia documentation on search A/B testing is technically adequate for getting started, and the dashboard has improved, but there are still many ways that you can go wrong when A/B testing Algolia search results. In the style of a 2014-era Buzzfeed listicle, here are 11 Algolia A/B Testing Gotchas, Tips, and Lessons!! All of the horrible illustrations are generated by AI, the rest are from earlier posts.

How Search and AI Product Teams are Different

harlan@harris.name (Harlan Harris) — Tue, 03 Feb 2026 00:00:00 +0000

Engineering leadership knows the standard playbook for product teams: the Product Trio, the Spotify model, outcome-based roadmaps, and so on. I've seen teams adopt these practices and still struggle when they add responsibility for advanced algorithms -- search, recommendations, predictive modeling, or generative AI -- without changing how the team is led. The processes and role expectations that work for a typical product team break down when the domain is heavily algorithmic. This post pulls together what I've learned and what others have written about building teams that own these systems.

Defensible Use of AI in Writing (Like This)

harlan@harris.name (Harlan Harris) — Thu, 29 Jan 2026 00:00:00 +0000

If AI helps me get ideas from my head to the readers' heads faster, that's good. Society thrives when ideas are shared, critiqued, and built on. But authors shouldn't take credit for other peoples' ideas -- they should synthesize others' ideas, including the "blurry JPG of the web" that is LLMs. The question is: what makes AI-assisted writing defensible? When is it helpful to society, and when is it plagiarism? When can you still be considered the author of AI-assisted writing?

Post Query Refinement Suggestions in Search UX, and an Algolia Demo App

harlan@harris.name (Harlan Harris) — Fri, 23 Jan 2026 00:00:00 +0000

What are Post-Query Refinement Suggestions?

One of my favorite search UX patterns is post-query refinement suggestions — buttons that appear between the search box and results, adjusting the query in various ways. See, for instance, these suggestions on Etsy, which recommend that I filter by shipping speed, seller type, cost range, style, etc.

Filter Suggestions on Etsy

PQRS is a mouthful of an abbreviation, but most of what I've seen as alternatives either describes the UI element ("refinement pills") rather than the function, or is excessively vague ("guided search"). So PQRSs it is, sorry.

Use LLMs for Translation and Fallible Reasoning

harlan@harris.name (Harlan Harris) — Wed, 21 Jan 2026 00:00:00 +0000

I keep seeing AI skeptic takes like "AI is useless" or "LLMs are only good at Natural Language Processing", usually because of hallucinations and AI slop. Both are real problems. But that conclusion still misses the point: it misidentifies what LLMs are actually good at.

My take is: LLMs are great at translation (in the broad sense), and they have emergent but fallible reasoning. Knowledge, in the "just tell me what's true" sense, is not their strong suit, despite how convincingly they can sound.

Vibe-Coding the Missing Algolia Comparison Dashboard

harlan@harris.name (Harlan Harris) — Tue, 13 Jan 2026 00:00:00 +0000

Algolia is a tremendously powerful search platform, especially for e-commerce. But like most SaaS tools, the management dashboard doesn't do everything you need it to do. In particular, it's hard to compare search result rankings across variations in indexes -- a super common pattern when you're testing and iterating on index configurations before A/B testing, implementing algorithm and UI proof-of-concepts, or refining search relevance.

At several places I've worked, I've built (or spec'ed out) comparison apps that allow you to run the same query in parallel and easily see and compare the results, for internal testing. Most recently, I vibe-coded an app that has worked very well. I can't share the code itself, so I've written a spec for a generic, equivalent app that is free for anybody to use and customize.

AI-Assisted Coding, Side Projects, and Donationware

harlan@harris.name (Harlan Harris) — Fri, 09 Jan 2026 00:00:00 +0000

The rise of AI-assisted coding tools has made it dramatically easier and faster to build small side projects than ever before. What used to take weeks of evenings and weekends can now be done in a few focused sessions, with an AI pair programmer handling much of the boilerplate, debugging, and implementation details. This has opened up new possibilities for building tools that solve personal problems or scratch specific itches, without the overhead of building a full commercial product.

A Collaborative Template for A/B Tests

harlan@harris.name (Harlan Harris) — Thu, 12 Oct 2023 00:00:00 +0000

As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we can learn from statistics. I (and many, many others) have found that there is a lot of value in having a document, shared among the team that is running the test. Some A/B testing tools include some limited workflow for collaboration on testing (e.g., Amplitude Experiments has pre-test, monitoring, and results tabs with useful graphs and summary statistics), but I think the most important thing is to have a tool that helps the team get aligned by forcing them to write things down.

Are LLMs a simple solution? And if so, for what problems?

harlan@harris.name (Harlan Harris) — Thu, 14 Sep 2023 00:00:00 +0000

The other day, I was reading a post by Venkatash Rao (thousands of words of under-edited brilliance, as usual), and was struck by this note about the complexity of climate solutions:

I tend to take as an article of faith the systems science rule of thumb that the complexity of solutions generally matches the complexity of the problems. If it doesn’t, then you either got lucky, or there are negative externalities you’re ignoring.

LLMs and Theories of Consciousness

harlan@harris.name (Harlan Harris) — Sun, 03 Sep 2023 00:00:00 +0000

There’s been an immense amount of discussion about Large Language Models (LLMs) such as ChatGPT over the last year, of course. Some of that discussion has been whether they are intelligent, conscious, or on the path to Artificial General Intelligence.

I’m particularly interested in the "consciousness" question, as it was an area of personal interest when I was working as a cognitive scientist, in a prior career. I never did research on the topic, but I read plenty of philosophers of mind and neuroscientists as they tried to pin down what, exactly (or even vaguely), consciousness might be. One of my favorite treatments of the topic is by neuroscientist Antonio Damasio, most accessibly covered in his book The Feeling of What Happens. (I recently read a new collection of short essays by him, entitled Feeling & Knowing, but I wouldn’t recommend it. It feels more like footnotes or stray thoughts than a coherent presentation or a novel contribution.)

The Five Types of A/B Test Decisions

harlan@harris.name (Harlan Harris) — Fri, 07 Jul 2023 00:00:00 +0000

Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it out.) Note that I go into a little more statistical detail here, although most of the below is readable by non-data scientists.

A Critique of The Markup’s Investigation into Predictive Models of Student Success

harlan@harris.name (Harlan Harris) — Mon, 29 May 2023 00:00:00 +0000

Recently, tech-journalism site The Markup ran a long, detailed, critical investigation of a predictive machine learning model used by the State of Wisconsin to identify public school students at risk of not graduating. I mostly agree with the conclusions of the piece -- the system appears not to be fit for purpose and needs to be substantially improved -- but I want to comment on several aspects of the model and the Markup’s reporting. Although I know nothing about the Wisconsin model beyond what is reported, I know a lot about predictive student success risk models, having led a team of data scientists who built related models used by colleges and universities when I worked at EAB from 2014 through 2016.

p Values Are Useful for A/B Tests, Sometimes

harlan@harris.name (Harlan Harris) — Tue, 02 May 2023 00:00:00 +0000

The "best practice", when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent 2016 statement that "...business... decisions should not be based only on whether a p-value passes a specific threshold." Wait, what? Are we making bad decisions from A/B tests? Should we stop using p values and do something else?

Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

harlan@harris.name (Harlan Harris) — Sat, 20 Aug 2022 00:00:00 +0000

A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various sorts – say, the proportion of people who visit your web site who buy at least one pair of shoes. Although the underlying data is just four numbers (visitors and purchasers, from the control group and the test group), there are lots of ways to compute statistics and present those numbers. Not all of those ways are ideal for supporting communication.

Staff Data Scientist: Comments on Will Larson's Staff Engineer Book

harlan@harris.name (Harlan Harris) — Fri, 27 May 2022 00:00:00 +0000

I recently read Will Larson's excellent book Staff Engineer: Leadership beyond the management track. Larson covers the individual contributor (IC, not management) roles that software engineers fill after they are promoted past Senior Software Engineer, with titles like Staff and Principal ("Staff-plus"). In the book, he synthesizes his own experience and the experiences of a number of other Staff-plus engineers, and provides great insights into how to get promoted to that level, and how to succeed at it. Great book -- you should read it.

A/B Testing and Product Ratings, Part 1: Delays and Bias

harlan@harris.name (Harlan Harris) — Sun, 22 May 2022 00:00:00 +0000

Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make the easy cases as simple as clicking a few buttons and looking at pretty graphs. But A/B tests can get statistically complex surprisingly quickly, which is why hiring data scientists with a strong statistics background can make a big difference in the quality of decisions.

New Publications and Upcoming Talks

harlan@harris.name (Harlan Harris) — Tue, 04 Sep 2018 00:00:00 +0000

Just a quick post here to note a few professional accomplishments:

I just added a new publication to my vita -- a peer-reviewed conference proceeedings article about abstractions for building repeated, related versions of similar predictive models. Check out some longer thoughts on Medium, or read the full article.
Earlier this year, I added an incredibly old project! A paper that I had contributed a bit to in... 2005! finally got published! It has something to do with the way information flows during speech perception... I think...
I'll be talking at two Meetups this Fall -- the RecSys NYC Meetup on Sept. 18th, and the Dataiku Data Science Meetup on Sept. 26th. In both Meetups, I'll be talking about (different) aspects of recommendations systems I'm building at WayUp.
And I'll be talking at two conferences this Fall too -- the Predictive APIs conference in Boston in October, and DataEngConf in NYC in November. At both of those conferences, I'll be talking about the software architecture aspects of building job recommendation systems that need to provide compelling recommendations just seconds after a user creates a rich profile.

On moving my blog from Wordpress to Blogdown

harlan@harris.name (Harlan Harris) — Sun, 11 Mar 2018 00:00:00 +0000

This is my first new post on harlan.harris.name for a while. The occasion is a change of scenery. For about 10 years, my primary blog has been on WordPress, more recently supplemented by Medium. But WordPress and Medium are limited for technical writing, and the trend among data people recently has been to publish static sites built with Blogdown and Hugo. So that’s what this is.

The technology I’m using (more on it below) lets me do fun things like trivially embed math: $\sum_i a^2_i$, or generate plots with embedded code:

On How and When to Teach Layers of Abstraction in Programming

harlan@harris.name (Harlan Harris) — Thu, 05 Oct 2017 00:00:00 +0000

This post was originally published on Medium

There’s recently been some interesting opinionated writing in the R statistical programming community about how and when to teach the abstracted, easy-to-use approaches to solving problems, versus the underlying nitty-gritty. David Robinson, Data Scientist at Stack Overflow, wrote a blog post recently called Don’t teach students the hard way first. The primary example was on the data-manipulation tools in the tidyverse, versus the underlying methods in base R, but the discussion was mostly about principles in pedagogy. Some highlight quotes from the original article (which I recommend reading!):

A few things I learned from two small Data Science conferences

harlan@harris.name (Harlan Harris) — Sun, 16 Jul 2017 00:00:00 +0000

This post was originally published on Medium

I recently attended two small conferences — the ISBIS (International Society for Business and Industrial Statistics) 2017 conference, held at IBM Research in Westchester County, and the Domino Data Lab Popup, held in West SoHo. I was invited to speak at ISBIS (slides here, if you’re curious), but for this post, I want to summarize some insights from other people’s talks.

In chronological (to me) order… First a few talks from ISBIS that I particularly liked (note that I only saw a fraction of all the talks):

What do Data Scientists mean by “Scaling”?

harlan@harris.name (Harlan Harris) — Fri, 09 Jun 2017 00:00:00 +0000

This post was originally published on Medium

Occasionally when chatting with other data scientists, especially with others who are interested in integrating predictive models into production software system, the word “scaling” comes up.

Not this. Although some West Coast data scientists are into this kind of scaling too.

I think this is a great question, but it’s a little underspecified. There seem to be at least three qualitatively different notions of “scaling” in data science, and it’s worth the effort to clarify each of them, and address how people tackle them.

Conference Blogging INFORMS Analytics 2017

harlan@harris.name (Harlan Harris) — Sun, 09 Apr 2017 00:00:00 +0000

This post was originally published on Medium

A particularly good way to get a little more out of professional conferences is to blog about your experiences, I think. It makes you focus your thoughts on things like “what’s the big take-away here,” and “what should I be asking people in the hallways?” Rather than just summarizing what you saw, or making snarky Twitter comments (also worth doing!), a great conference blog post is synthesis — combining insights from multiple presentations and conversations into a coherent new whole that helps clarify ideas.

Transparency, Trust, and Proprietary Predictive Analytics

harlan@harris.name (Harlan Harris) — Mon, 27 Feb 2017 00:00:00 +0000

This post was originally published on Medium

A particularly good talk at Strata NY last year was by Brett Goldstein, former CIO of Chicago, who talked about accountability and transparency in predictive models that affect people’s lives. This struck a strong chord with me, so I wanted to take some time to write down some thoughts. (And a rather longer time to publish those thoughts…) I’m sure others’ have thought about this more and have better takes on this — please comment and provide links!

neveragain.tech

harlan@harris.name (Harlan Harris) — Sun, 08 Jan 2017 00:00:00 +0000

I, Harlan D. Harris, hereby commit to the neveragain.tech pledge. Please stand with me and hold me to it.

It starts:

We, the undersigned, are employees of tech organizations and companies based in the United States. We are engineers, designers, business executives, and others whose jobs include managing or processing data about people. We are choosing to stand in solidarity with Muslim Americans, immigrants, and all people whose lives and livelihoods are threatened by the incoming administration’s proposed data collection policies. We refuse to build a database of people based on their Constitutionally-protected religious beliefs. We refuse to facilitate mass deportations of people the government believes to be undesirable.

(read the rest)

Insights from a Predictive Model Pipeline Abstraction

harlan@harris.name (Harlan Harris) — Mon, 07 Nov 2016 00:00:00 +0000

This post was originally published on Medium

When building a complex system, it’s often helpful to think about the design of that system using patterns and abstractions. Architects and software engineers do so frequently, and the experience of implementing predictive modeling pipelines has recently led to a variety of patterns and best practices. For instance, when dealing with large amounts of streaming data, some organizations use the Lambda Architecture to handle both real-time and computationally-intensive use-cases.

Patterns for Connecting Predictive Models to Software Products

harlan@harris.name (Harlan Harris) — Tue, 12 Jul 2016 00:00:00 +0000

This post was originally published on Medium

You’re a data scientist, and you’ve got a predictive model — great work! Now what? In many cases, you need to hook it up to some sort of large, complex software product so that users can get access to the predictions. Think of LinkedIn’s People You May Know, which mines your professional graph for unconnected connections, or Hopper’s flight price predictions. Those started out as prototypes on someone’s laptop, and are now running at scale, with many millions of users.

Simulating Rent Stabilization Policy at the National Day of Civic Hacking

harlan@harris.name (Harlan Harris) — Sun, 05 Jun 2016 00:00:00 +0000

This post was originally published on Medium

Yesterday was the 2016 National Day of Civic Hacking, a Code for America event that encourages people with technology and related skills to explore projects related to civil society and government. My friend Josh Tauberer wrote a thoughtful post earlier about the event called Why We Hack —on what the value of this sort of event might be — please read it.

For my part, this year I worked on one of the projects he discusses, understanding the impact of DC’s rent stabilization laws and what potential policy changes might yield. As Josh noted, we discovered that it’s a hard problem. Much of the most relevant data (such as the list of properties under rent stabilization and their current and historical rents) are not available, and have to be estimated. Getting to a realistic understanding of the impact of law and policy on rents seems incredibly valuable, but hard.

Thoughts on Managing Data Science Team Workstreams (and a Shiny app)

harlan@harris.name (Harlan Harris) — Thu, 28 Jan 2016 00:00:00 +0000

This post was originally published on Medium

View this post on X (josh_wills)

There are different types of data scientists, with different backgrounds and career paths. With Sean Murphy and Marck Vaisman, I wrote an article about this for O’Reilly a few years back, based on survey research we’d done. Download a copy, if you haven’t read it. This idea is now pretty well established, but I want to talk about a related issue, which is that the type of workthat Data Science teams do varies a lot, and that managing those types of work can be an interesting challenge.

Building a Complementary Data Science Team

harlan@harris.name (Harlan Harris) — Mon, 23 Nov 2015 00:00:00 +0000

This post was originally published on Medium

I’m the Director of Data Science at EAB, a firm that provides best-practices research and enterprise software for colleges and universities. My team is responsible for the predictive models and other advanced analytics that are part of the Student Success Collaborative product that’s used by academic advisors and other campus leadership. We’re hiring data scientists, and I wanted to publicly say a few things about the roles we have advertised. (Note that EAB is part of a public company and is in a competitive market, so there are obviously things I’m not saying!)

Parameterizable Reproducible Research

harlan@harris.name (Harlan Harris) — Thu, 20 Nov 2014 00:00:00 +0000

The below is a public version of a post originally posted on an internal blog at the Education Advisory Board (EAB), my current employer. We don’t yet have a public tech blog, but I got permission to edit and post it here, along with the referenced code.

Data Science teams get asked to do a lot of different sorts of things. Some of what the team that I’m part of builds is enterprise-scale predictive analytics, such as the Student Risk Model that’s part of the Student Success Collaborative. That’s basically software development with a statistical twist and machine-learning core. Sometimes we get asked to do quick-and-dirty, one-off sorts of things, to answer a research question. We have a variety of tools and processes for that task. But there’s a third category that I want to focus on – frequently requested but slightly-different reports.

what is it

There’s a relatively new theme in the scientific research community called reproducible research. Briefly, the idea is that it should be possible to re-do all steps after data collection automatically, including data cleaning and reformatting, statistical analyses, and even the actual generation of a camera-ready report with charts, graphs, and tables. This means that if you realized that, say, one data point in your analysis was bogus and needed to be removed, you could remove that data point, press a button, and in a minute or two have a shiny new PDF with all of the results automatically updated.

This type of reproducible research has been around for a while, although it’s having a recent resurgence in part due to the so-called “statistical crisis”. The R (and S) statistical programming languages have supported LaTeX, the scientific document creation/typesetting tool, for many years. Using a tool called Sweave, a researcher “weaves” chunks of text and chunks of R code together. The document is then “executed”, where the R code chunks are executed and the results are converted into a single LaTeX document, which is then compiled into a PDF or similar. The code can generate charts and tables, so no manual effort is needed to rebuild a camera-ready document.

This is great, a huge step forward towards validation of often tricky and complex statistical analyses. If you’re writing a conference paper on, say, a biomedical experiment, a reproducible process can drastically improve your ability to be confident in your work. But data scientists often have to generate this sort of thing repeatedly, from different sources of data or with different parameters. And they have to do so efficiently.

Parameterizable reproducible research, then, is a variant of reproducible research tools and workflows where it is easy to specify data sources, options, and parameters to a standardized analytical report, even one that includes statistical or predictive analyses, data manipulation, and graph generation. The report can be emailed or otherwise sent to people, and doesn’t seem as public as, say, a web-based app developed in Shiny or another technology. This isn’t a huge breakthrough or anything, but it’s a useful pattern that seems worth sharing.

INFORMS Business Analytics 2014 Blog Posts

harlan@harris.name (Harlan Harris) — Sat, 02 Aug 2014 00:00:00 +0000

Earlier this year, I attended the INFORMS Conference on Business Analytics & Operations Research, in Boston. I was asked beforehand if I wanted to be a conference blogger, and for some reason I said I would. This meant I was able to publish posts on the conference’s WordPress web site, and was also obliged to do so!

Here are the five posts that I wrote, along with an excerpt from each. Please click through to read the full pieces:

Operations Research, from the point of view of Data Science

more insight, less action — deliverables tend towards predictions and storytelling, versus formal optimization
more openness, less big iron — open source software leads to a low-cost, highly flexible approach
more scruffy, less neat — data science technologies often come from black-box statistical models, vs. domain-based theory
more velocity, smaller projects — a hundred $10K projects beats one $1M project
more science, less engineering — both practitioners and methods have different backgrounds
more hipsters, less suits — stronger connections to the tech industry than to the boardroom
more rockstars, less teams — one person can now (roughly) do everything, in simple cases, for better or worse

What is a “Data Product”?

DJ Patil says “a data product is a product that facilitates an end goal through the use of data.” So, it’s not just an analysis, or a recommendation to executives, or an insight that leads to an improvement to a business process. It’s a visible component of a system. LinkedIn’s People You May Know is viewed by many millions of customers, and it’s based on the complex interactions of the customers themselves.

Why a Data Community is Like a Music Scene -- Resources

harlan@harris.name (Harlan Harris) — Sat, 26 Oct 2013 00:00:00 +0000

On Monday, October 28th, 2013, I gave a 5-minute Ignite talk entitled “Why a Data Community is Like a Music Scene” at an event associated with the Strata conference. Here’s the video:

And here are the acknowledgements and references for the talk:

Data Community DC

How Music Works, by David Byrne

my slides for the Ignite talk

my blog post (written first)

Photos:

CBGB’s exterior: NYC - East Village: CBGB & OMFUG by wallyg, on Flickr (Creative Commons)
Grafitti wall: cbgb, september 2006 (#1) by joe holmes, on Flickr (Creative Commons)
Joan Jett with beer: 1977 Los Angeles, CA-Joan Jett of the Runaways backstage at The Whiskey A Go Go. (Photos by Brad Elterman/BuzzFoto.com)
CSN: legendsrevealed.com
Flats to Let: skyscrapercity.com
Dressing room: CBGBS Dresssing Room by Chris Infidel, on Flickr
Snog: _MG_1977.jpg by dinoboy, on Flickr

Communication and the Data Scientist

harlan@harris.name (Harlan Harris) — Tue, 23 Oct 2012 00:00:00 +0000

I recently gave a presentation on communication issues around the terms “Data Science” and “Data Scientist”, based in part on a survey that I did with my Meetup colleagues Marck and Sean. The basic idea is that these new, extremely-broad buzzwords have resulted in confusion, which has impacted the ability of people with skills and people with data to meet and effectively communicate about who does what and what appropriate expectations should be. The survey was an attempt to bring some clarity to the issue of who are the people in this newly-reformulated community, and how do they view themselves and their skills. For more on the survey, see our post on the Data Community DC blog. Here’s the video of my presentation at DataGotham:

integrating R with other systems

harlan@harris.name (Harlan Harris) — Sat, 16 Jun 2012 00:00:00 +0000

I just returned from the useR! 2012 conference for developers and users of R. One of the common themes to many of the presentations was integration of R-based statistical systems with other systems, be they other programming languages, web systems, or enterprise data systems. Some highlights for me were an update to Rserve that includes 1-stop web services, and a presentation on ESB integration. Although I didn’t see it discussed, the new httr package for easier access to web services is also another outstanding development in integrating R into large-scale systems.

Survey of Data Science / Analytics / Big Data / Applied Stats / Machine Learning etc. Practitioners

harlan@harris.name (Harlan Harris) — Thu, 10 May 2012 00:00:00 +0000

As I’ve discussed here before, there is a debate raging (ok, maybe not raging) about terms such as “data science”, “analytics”, “data mining”, and “big data”. What do they mean, how do they overlap, and perhaps most importantly, who are the people who work in these fields?

Along with two other DC-area Data Scientists, Marck Vaisman and Sean Murphy, I’ve put together a survey to explore some of these issues. Help us quantitatively understand the space of data-related skills and careers by participating!

Data Science, Moore's Law, and Moneyball

harlan@harris.name (Harlan Harris) — Tue, 27 Sep 2011 00:00:00 +0000

I’m fond of navel gazing, meta discussions, and so forth. I’ve recently written about inferring navel gazing from link data, and about the meaning of the “Analytics” buzzword. This post will be my second on that other infectious buzzword, “Data Science”.

When I moved to Washington DC in July, I was struck by the fact that there was no Meetup for analytics/applied statistics/machine learning/data science. There’s a great DC Tech Meetup, a great Big Data Meetup, and a great R Meetup, but nothing like the NYC Predictive Analytics Meetup. So, I and a couple of others I talked to about this (Marck Vaisman, who I first met through the NYC R Meetup a couple years ago, and Matt Bryan, who I met just after moving to town), started a new Meetup, which we decided to call “Data Science DC”.

hacking .gov shortened links

harlan@harris.name (Harlan Harris) — Sat, 30 Jul 2011 00:00:00 +0000

This past Friday, the web portal to the US Federal government, USA.gov, organized hackathons across the US for programmers and data scientists to work with and analyze the data from their link-shortening service. It turns out that if you shorten a web link with bit.ly, the shortened link looks like 1.usa.gov/V6NpL (that one goes to a NASA page). And because this service was paid for by taxpayer money, the data about each clickthrough is freely available.

making meat shares more efficient with R and Symphony

harlan@harris.name (Harlan Harris) — Mon, 09 May 2011 01:01:01 +0000

In my previous post, I motivated a web application that would allow small-scale sustainable meat producers to sell directly to consumers using a meat share approach, using constrained optimization techniques to maximize utility for everyone involved. In this post, I’ll walk through some R code that I wrote to demonstrate the technique on a small scale.

Although the problem is set up in R, the actual mathematical optimization is done by Symphony, an open-source mixed-integer solver that’s part of the COIN-OR project. (The problem of optimizing assignments, in this case of cuts of meat to people, is an integer planning problem, because the solution involves assigning either 0 or 1 of each cut to each person. More generally, linear programming and related optimization frameworks allow solving for real-numbered variables.) The RSymphony package allows problems set up in R to be solved by the C/C++ Symphony code with little hassle.

making meat shares more efficient

harlan@harris.name (Harlan Harris) — Mon, 09 May 2011 00:00:00 +0000

A personal interest I have is the ethical and sustainable production of food. I’ve been a member of and helped run Community Supported Agriculture groups, and my wife and I currently purchase the majority of our meat from a group of upstate NY pastured-livestock producers who sell their products through CSAs. It’s an ala-carte business model, where I place an order on a website, and the next week I pick up the frozen products cut and packaged as if for retail.

intuitive visualizations of categorization for non-technical audiences

harlan@harris.name (Harlan Harris) — Mon, 25 Apr 2011 00:00:00 +0000

For a project I’m working on at work, I’m building a predictive model that categorizes something (I can’t tell you what) into two bins. There is a default bin that 95% of the things belong to and a bin that the business cares a lot about, containing 5% of the things. Some readers may be familiar with the use of predictive models to identify better sales leads, so that you can target the leads most likely to convert and minimize the amount of effort wasted on people who won’t purchase your product. Although my situation doesn’t have to do with sales leads, I’m going to pretend it does, as it’s a common domain.

My data is many thousands of “leads”, for which I’ve constructed hundreds of predictive features (mostly 1/0, a few numeric) each. I can plug this data into any number of common statistical and machine learning systems which will crunch the numbers and provide a black box that can do a pretty good job of separating more-valuable leads from less valuable leads. That’s great, but now I have to communicate what I’ve done, and how valuable it is, to an audience that struggles with relatively simple statistical concepts like correlation. What can I do?

On "Analytics" and related fields

harlan@harris.name (Harlan Harris) — Fri, 15 Apr 2011 00:00:00 +0000

I recently attended the INFORMS Conference on Business Analytics and Operations Research, aka “INFORMS Analytics 2011”, conference in Chicago. This deserves a little bit of an explanation. INFORMS is the professional organization for Operations Research (OR) and Management Science (MS), which are terms describing approaches to improving business efficiency by use of mathematical optimization and simulation tools. OR is perhaps best known for the technique of Linear Programming (read “Programming” as “Planning”), which is a method for optimizing a useful class of mathematical expressions under various constraints extremely efficiently. You can, for example, solve scheduling, assignment, transportation, factory layout, and similar problems with millions of variables in seconds. These techniques came out of large-scale government and especially military logistics and decision-making needs of the mid-20th century, and have now been applied extensively in many industries. Have you seen the UPS “We (heart) Logistics” ad? That’s OR.

"Data Scientist" and other titles

harlan@harris.name (Harlan Harris) — Sun, 13 Feb 2011 00:00:00 +0000

Neil Saunders has an interesting (to me) blog post up this morning, with the title “Dumped on by data scientists.” He uses the use of “data scientist” in a Chronicle of Higher Ed article to rant a little bit about the term. For Neil, it’s redundant, as the act of doing science necessarily requires data; it’s insulting, as if “scientist” wasn’t cool enough and you have to add “data”; and it’s misleading, as many people who call themselves “data scientists” are actually dealing with business data rather than scientific data.

how to speak ggplot2 like a native, and Predictive Analytics World

harlan@harris.name (Harlan Harris) — Sun, 24 Oct 2010 00:00:00 +0000

I was recently given the opportunity to re-present my ggplot2 talk, which I originally gave to the NYC R Meetup, to the DC R Meetup group. The Meetup was held co-located with the Predictive Analytics World conference in Alexandria, VA. (More on my thoughts on PAW below…) Contentwise, I made only small changes, changing a bit of patter and adding more examples at the end. I still love ggplot, with some frustration at the way it is typically introduced. Some of the audience had no R experience at all, while others were experts. One person, a grad student at U. of Maryland, had had very similar difficulty as I had when originally learning ggplot2, and his enthusiastic nods during my presentation were very validating! For reference, the Meetup page is here, and I stuck the current version of the slides in a public Dropbox, located here.

Prediction with Multilevel Regression Models, and Pizza

harlan@harris.name (Harlan Harris) — Fri, 15 Oct 2010 00:00:00 +0000

The Meetup phenomenon, which is now substantial and longstanding enough to be more of a cultural change than a flash in the pan, continues to impress me. Even more so than tools like LinkedIn, Meetups have changed the nature of professional networking, making it more informal, diverse, and decentralized. Last night, statistics consultant (and cheap eats guru) Jared Lander and I presented a talk on a statistical technique tangentially related to my professional work (more closely associated with Jared’s). The origin of this presentation is worth noting. On Meetup’s web site, members of a group can suggest topics for meetings. Before even attending a single NYC Predictive Analytics event, I posted several topics that I thought might be interesting for the group. A bit later, the organizers (Bruno and Alex) contacted me to see if I’d be willing to present on prediction with Multilevel models. I said that I would, but only if I could co-present with someone who actually knew something about the topic a complementary set of skills and experiences. Knowing Jared from the NYC R Meetup group, and knowing that he learned about multilevel models from the professor who wrote the best book on the topic, and knowing that he’s pretty good in front of an audience, I suggested we collaborate.

ggplot and concepts -- what's right, and what's wrong

harlan@harris.name (Harlan Harris) — Sun, 07 Mar 2010 00:00:00 +0000

A few months back I gave a presentation to the NYC R Meetup. (R is a statistical programming language. If this means nothing to you, feel free to stop reading now.) The presentation was on ggplot2, a popular package for generating graphs of data and statistics. In the talk (which you can see here, including both my slides and my patter!) I presented both the really great things about ggplot2 and some of its downsides. In this blog post, I wanted to expand a bit on my thinking on ggplot, the Grammar of Graphics, and how peoples’ conceptual representations of graphs, data, ggplot, and R all interact. ggplot is both incredibly elegant and unfortunately difficult to learn to use well, I think as a consequence of the variety of representations.

Online publishing, micropayments, and warm fuzzy feelings

harlan@harris.name (Harlan Harris) — Tue, 20 Oct 2009 00:00:00 +0000

The problem of how to monetize online publishing, particularly news publishing, is neither new nor all that surprising. But the ongoing lack of a solution is steadily eating into news organizations across the country. Yesterday, the Times announced it was going to buy out or lay off 8% of its newsroom staff, despite being the best national newspaper in the country and probably the one making the best use of Internet technologies. (Their interactive graphics are some of the best around.) How can newspapers make money on the web? Ad revenue is inadequate, and people won’t generally pay for content. This post from a journalism blog at Harvard discusses why micropayments will never work:

Professional on Harlan D. Harris

A Big Win from a Small Search UI Change and a Predictive Model

Design Decisions for a New Search Results Page

You Don't Need a Search Engineer; You Need a Search Scientist

Nine Things I Learned Building a Scrabble App for My Friends

IZE meets AI -- semantic search, smarter labels, and agentic orientation

GeneralIZE -- How else could IZE's hierarchies be generated?

Could We Build IZE Again?

What Came After IZE? Three Domains, Three Answers

How IZE Really Worked - Algorithm, Patent, Limits, and Esther Dyson

The algorithm

IZE - Revisiting a hierarchical search technology from the PC era

11 Algolia A/B Testing Gotchas, Tips, and Lessons!!

How Search and AI Product Teams are Different

Defensible Use of AI in Writing (Like This)

Post Query Refinement Suggestions in Search UX, and an Algolia Demo App

What are Post-Query Refinement Suggestions?

Filter Suggestions on Etsy

Use LLMs for Translation and Fallible Reasoning

Vibe-Coding the Missing Algolia Comparison Dashboard

AI-Assisted Coding, Side Projects, and Donationware

A Collaborative Template for A/B Tests

Are LLMs a simple solution? And if so, for what problems?

LLMs and Theories of Consciousness

The Five Types of A/B Test Decisions

A Critique of The Markup’s Investigation into Predictive Models of Student Success

p Values Are Useful for A/B Tests, Sometimes

Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

Staff Data Scientist: Comments on Will Larson's Staff Engineer Book

A/B Testing and Product Ratings, Part 1: Delays and Bias

New Publications and Upcoming Talks

On moving my blog from Wordpress to Blogdown

On How and When to Teach Layers of Abstraction in Programming

A few things I learned from two small Data Science conferences

What do Data Scientists mean by “Scaling”?

Conference Blogging INFORMS Analytics 2017

Transparency, Trust, and Proprietary Predictive Analytics

neveragain.tech

Insights from a Predictive Model Pipeline Abstraction

Patterns for Connecting Predictive Models to Software Products

Simulating Rent Stabilization Policy at the National Day of Civic Hacking

Thoughts on Managing Data Science Team Workstreams (and a Shiny app)

Building a Complementary Data Science Team

Parameterizable Reproducible Research

what is it

INFORMS Business Analytics 2014 Blog Posts

Operations Research, from the point of view of Data Science

What is a “Data Product”?

Why a Data Community is Like a Music Scene -- Resources

More posts on the Data Community DC blog

Communication and the Data Scientist

integrating R with other systems

Survey of Data Science / Analytics / Big Data / Applied Stats / Machine Learning etc. Practitioners

Data Science, Moore's Law, and Moneyball

hacking .gov shortened links

making meat shares more efficient with R and Symphony

making meat shares more efficient

intuitive visualizations of categorization for non-technical audiences

On "Analytics" and related fields

"Data Scientist" and other titles

how to speak ggplot2 like a native, and Predictive Analytics World

Prediction with Multilevel Regression Models, and Pizza

ggplot and concepts -- what's right, and what's wrong

Online publishing, micropayments, and warm fuzzy feelings