Data-Science on Harlan D. Harris

You Don't Need a Search Engineer; You Need a Search Scientist

harlan.harris@gmail.com (Harlan Harris) — Mon, 13 Apr 2026 00:00:00 +0000

Say you're an Engineering Manager or a Product Manager, responsible for a business-critical search system on a website. Maybe e-commerce, maybe a marketplace, maybe something else. Search is sort of working, it doesn't crash, and it's fast, but there's a feeling that "search isn't great", and you've been tasked with finding someone to help. Your first instinct, or that of your VP, might be some version of: “we need to hire a search engineer.”

A Collaborative Template for A/B Tests

harlan.harris@gmail.com (Harlan Harris) — Thu, 12 Oct 2023 00:00:00 +0000

As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we can learn from statistics. I (and many, many others) have found that there is a lot of value in having a document, shared among the team that is running the test. Some A/B testing tools include some limited workflow for collaboration on testing (e.g., Amplitude Experiments has pre-test, monitoring, and results tabs with useful graphs and summary statistics), but I think the most important thing is to have a tool that helps the team get aligned by forcing them to write things down.

The Five Types of A/B Test Decisions

harlan.harris@gmail.com (Harlan Harris) — Fri, 07 Jul 2023 00:00:00 +0000

Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it out.) Note that I go into a little more statistical detail here, although most of the below is readable by non-data scientists.

p Values Are Useful for A/B Tests, Sometimes

harlan.harris@gmail.com (Harlan Harris) — Tue, 02 May 2023 00:00:00 +0000

The "best practice", when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent 2016 statement that "...business... decisions should not be based only on whether a p-value passes a specific threshold." Wait, what? Are we making bad decisions from A/B tests? Should we stop using p values and do something else?

Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

harlan.harris@gmail.com (Harlan Harris) — Sat, 20 Aug 2022 00:00:00 +0000

A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various sorts – say, the proportion of people who visit your web site who buy at least one pair of shoes. Although the underlying data is just four numbers (visitors and purchasers, from the control group and the test group), there are lots of ways to compute statistics and present those numbers. Not all of those ways are ideal for supporting communication.

Staff Data Scientist: Comments on Will Larson's Staff Engineer Book

harlan.harris@gmail.com (Harlan Harris) — Fri, 27 May 2022 00:00:00 +0000

I recently read Will Larson's excellent book Staff Engineer: Leadership beyond the management track. Larson covers the individual contributor (IC, not management) roles that software engineers fill after they are promoted past Senior Software Engineer, with titles like Staff and Principal ("Staff-plus"). In the book, he synthesizes his own experience and the experiences of a number of other Staff-plus engineers, and provides great insights into how to get promoted to that level, and how to succeed at it. Great book -- you should read it.

A/B Testing and Product Ratings, Part 1: Delays and Bias

harlan.harris@gmail.com (Harlan Harris) — Sun, 22 May 2022 00:00:00 +0000

Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make the easy cases as simple as clicking a few buttons and looking at pretty graphs. But A/B tests can get statistically complex surprisingly quickly, which is why hiring data scientists with a strong statistics background can make a big difference in the quality of decisions.

New Publications and Upcoming Talks

harlan.harris@gmail.com (Harlan Harris) — Tue, 04 Sep 2018 00:00:00 +0000

Just a quick post here to note a few professional accomplishments:

I just added a new publication to my vita -- a peer-reviewed conference proceeedings article about abstractions for building repeated, related versions of similar predictive models. Check out some longer thoughts on Medium, or read the full article.
Earlier this year, I added an incredibly old project! A paper that I had contributed a bit to in... 2005! finally got published! It has something to do with the way information flows during speech perception... I think...
I'll be talking at two Meetups this Fall -- the RecSys NYC Meetup on Sept. 18th, and the Dataiku Data Science Meetup on Sept. 26th. In both Meetups, I'll be talking about (different) aspects of recommendations systems I'm building at WayUp.
And I'll be talking at two conferences this Fall too -- the Predictive APIs conference in Boston in October, and DataEngConf in NYC in November. At both of those conferences, I'll be talking about the software architecture aspects of building job recommendation systems that need to provide compelling recommendations just seconds after a user creates a rich profile.

A few things I learned from two small Data Science conferences

harlan.harris@gmail.com (Harlan Harris) — Sun, 16 Jul 2017 00:00:00 +0000

This post was originally published on Medium

I recently attended two small conferences — the ISBIS (International Society for Business and Industrial Statistics) 2017 conference, held at IBM Research in Westchester County, and the Domino Data Lab Popup, held in West SoHo. I was invited to speak at ISBIS (slides here, if you’re curious), but for this post, I want to summarize some insights from other people’s talks.

In chronological (to me) order… First a few talks from ISBIS that I particularly liked (note that I only saw a fraction of all the talks):

What do Data Scientists mean by “Scaling”?

harlan.harris@gmail.com (Harlan Harris) — Fri, 09 Jun 2017 00:00:00 +0000

This post was originally published on Medium

Occasionally when chatting with other data scientists, especially with others who are interested in integrating predictive models into production software system, the word “scaling” comes up.

Not this. Although some West Coast data scientists are into this kind of scaling too.

I think this is a great question, but it’s a little underspecified. There seem to be at least three qualitatively different notions of “scaling” in data science, and it’s worth the effort to clarify each of them, and address how people tackle them.

Conference Blogging INFORMS Analytics 2017

harlan.harris@gmail.com (Harlan Harris) — Sun, 09 Apr 2017 00:00:00 +0000

This post was originally published on Medium

A particularly good way to get a little more out of professional conferences is to blog about your experiences, I think. It makes you focus your thoughts on things like “what’s the big take-away here,” and “what should I be asking people in the hallways?” Rather than just summarizing what you saw, or making snarky Twitter comments (also worth doing!), a great conference blog post is synthesis — combining insights from multiple presentations and conversations into a coherent new whole that helps clarify ideas.

Transparency, Trust, and Proprietary Predictive Analytics

harlan.harris@gmail.com (Harlan Harris) — Mon, 27 Feb 2017 00:00:00 +0000

This post was originally published on Medium

A particularly good talk at Strata NY last year was by Brett Goldstein, former CIO of Chicago, who talked about accountability and transparency in predictive models that affect people’s lives. This struck a strong chord with me, so I wanted to take some time to write down some thoughts. (And a rather longer time to publish those thoughts…) I’m sure others’ have thought about this more and have better takes on this — please comment and provide links!

Insights from a Predictive Model Pipeline Abstraction

harlan.harris@gmail.com (Harlan Harris) — Mon, 07 Nov 2016 00:00:00 +0000

This post was originally published on Medium

When building a complex system, it’s often helpful to think about the design of that system using patterns and abstractions. Architects and software engineers do so frequently, and the experience of implementing predictive modeling pipelines has recently led to a variety of patterns and best practices. For instance, when dealing with large amounts of streaming data, some organizations use the Lambda Architecture to handle both real-time and computationally-intensive use-cases.

Patterns for Connecting Predictive Models to Software Products

harlan.harris@gmail.com (Harlan Harris) — Tue, 12 Jul 2016 00:00:00 +0000

This post was originally published on Medium

You’re a data scientist, and you’ve got a predictive model — great work! Now what? In many cases, you need to hook it up to some sort of large, complex software product so that users can get access to the predictions. Think of LinkedIn’s People You May Know, which mines your professional graph for unconnected connections, or Hopper’s flight price predictions. Those started out as prototypes on someone’s laptop, and are now running at scale, with many millions of users.

Thoughts on Managing Data Science Team Workstreams (and a Shiny app)

harlan.harris@gmail.com (Harlan Harris) — Thu, 28 Jan 2016 00:00:00 +0000

This post was originally published on Medium

View this post on X (josh_wills)

There are different types of data scientists, with different backgrounds and career paths. With Sean Murphy and Marck Vaisman, I wrote an article about this for O’Reilly a few years back, based on survey research we’d done. Download a copy, if you haven’t read it. This idea is now pretty well established, but I want to talk about a related issue, which is that the type of workthat Data Science teams do varies a lot, and that managing those types of work can be an interesting challenge.

Building a Complementary Data Science Team

harlan.harris@gmail.com (Harlan Harris) — Mon, 23 Nov 2015 00:00:00 +0000

This post was originally published on Medium

I’m the Director of Data Science at EAB, a firm that provides best-practices research and enterprise software for colleges and universities. My team is responsible for the predictive models and other advanced analytics that are part of the Student Success Collaborative product that’s used by academic advisors and other campus leadership. We’re hiring data scientists, and I wanted to publicly say a few things about the roles we have advertised. (Note that EAB is part of a public company and is in a competitive market, so there are obviously things I’m not saying!)

INFORMS Business Analytics 2014 Blog Posts

harlan.harris@gmail.com (Harlan Harris) — Sat, 02 Aug 2014 00:00:00 +0000

Earlier this year, I attended the INFORMS Conference on Business Analytics & Operations Research, in Boston. I was asked beforehand if I wanted to be a conference blogger, and for some reason I said I would. This meant I was able to publish posts on the conference’s WordPress web site, and was also obliged to do so!

Here are the five posts that I wrote, along with an excerpt from each. Please click through to read the full pieces:

Operations Research, from the point of view of Data Science

more insight, less action — deliverables tend towards predictions and storytelling, versus formal optimization
more openness, less big iron — open source software leads to a low-cost, highly flexible approach
more scruffy, less neat — data science technologies often come from black-box statistical models, vs. domain-based theory
more velocity, smaller projects — a hundred $10K projects beats one $1M project
more science, less engineering — both practitioners and methods have different backgrounds
more hipsters, less suits — stronger connections to the tech industry than to the boardroom
more rockstars, less teams — one person can now (roughly) do everything, in simple cases, for better or worse

What is a “Data Product”?

DJ Patil says “a data product is a product that facilitates an end goal through the use of data.” So, it’s not just an analysis, or a recommendation to executives, or an insight that leads to an improvement to a business process. It’s a visible component of a system. LinkedIn’s People You May Know is viewed by many millions of customers, and it’s based on the complex interactions of the customers themselves.

Why a Data Community is Like a Music Scene -- Resources

harlan.harris@gmail.com (Harlan Harris) — Sat, 26 Oct 2013 00:00:00 +0000

On Monday, October 28th, 2013, I gave a 5-minute Ignite talk entitled “Why a Data Community is Like a Music Scene” at an event associated with the Strata conference. Here’s the video:

And here are the acknowledgements and references for the talk:

Data Community DC

How Music Works, by David Byrne

my slides for the Ignite talk

my blog post (written first)

Photos:

CBGB’s exterior: NYC - East Village: CBGB & OMFUG by wallyg, on Flickr (Creative Commons)
Grafitti wall: cbgb, september 2006 (#1) by joe holmes, on Flickr (Creative Commons)
Joan Jett with beer: 1977 Los Angeles, CA-Joan Jett of the Runaways backstage at The Whiskey A Go Go. (Photos by Brad Elterman/BuzzFoto.com)
CSN: legendsrevealed.com
Flats to Let: skyscrapercity.com
Dressing room: CBGBS Dresssing Room by Chris Infidel, on Flickr
Snog: _MG_1977.jpg by dinoboy, on Flickr

Communication and the Data Scientist

harlan.harris@gmail.com (Harlan Harris) — Tue, 23 Oct 2012 00:00:00 +0000

I recently gave a presentation on communication issues around the terms “Data Science” and “Data Scientist”, based in part on a survey that I did with my Meetup colleagues Marck and Sean. The basic idea is that these new, extremely-broad buzzwords have resulted in confusion, which has impacted the ability of people with skills and people with data to meet and effectively communicate about who does what and what appropriate expectations should be. The survey was an attempt to bring some clarity to the issue of who are the people in this newly-reformulated community, and how do they view themselves and their skills. For more on the survey, see our post on the Data Community DC blog. Here’s the video of my presentation at DataGotham:

integrating R with other systems

harlan.harris@gmail.com (Harlan Harris) — Sat, 16 Jun 2012 00:00:00 +0000

I just returned from the useR! 2012 conference for developers and users of R. One of the common themes to many of the presentations was integration of R-based statistical systems with other systems, be they other programming languages, web systems, or enterprise data systems. Some highlights for me were an update to Rserve that includes 1-stop web services, and a presentation on ESB integration. Although I didn’t see it discussed, the new httr package for easier access to web services is also another outstanding development in integrating R into large-scale systems.

Survey of Data Science / Analytics / Big Data / Applied Stats / Machine Learning etc. Practitioners

harlan.harris@gmail.com (Harlan Harris) — Thu, 10 May 2012 00:00:00 +0000

As I’ve discussed here before, there is a debate raging (ok, maybe not raging) about terms such as “data science”, “analytics”, “data mining”, and “big data”. What do they mean, how do they overlap, and perhaps most importantly, who are the people who work in these fields?

Along with two other DC-area Data Scientists, Marck Vaisman and Sean Murphy, I’ve put together a survey to explore some of these issues. Help us quantitatively understand the space of data-related skills and careers by participating!

Data Science, Moore's Law, and Moneyball

harlan.harris@gmail.com (Harlan Harris) — Tue, 27 Sep 2011 00:00:00 +0000

I’m fond of navel gazing, meta discussions, and so forth. I’ve recently written about inferring navel gazing from link data, and about the meaning of the “Analytics” buzzword. This post will be my second on that other infectious buzzword, “Data Science”.

When I moved to Washington DC in July, I was struck by the fact that there was no Meetup for analytics/applied statistics/machine learning/data science. There’s a great DC Tech Meetup, a great Big Data Meetup, and a great R Meetup, but nothing like the NYC Predictive Analytics Meetup. So, I and a couple of others I talked to about this (Marck Vaisman, who I first met through the NYC R Meetup a couple years ago, and Matt Bryan, who I met just after moving to town), started a new Meetup, which we decided to call “Data Science DC”.

On "Analytics" and related fields

harlan.harris@gmail.com (Harlan Harris) — Fri, 15 Apr 2011 00:00:00 +0000

I recently attended the INFORMS Conference on Business Analytics and Operations Research, aka “INFORMS Analytics 2011”, conference in Chicago. This deserves a little bit of an explanation. INFORMS is the professional organization for Operations Research (OR) and Management Science (MS), which are terms describing approaches to improving business efficiency by use of mathematical optimization and simulation tools. OR is perhaps best known for the technique of Linear Programming (read “Programming” as “Planning”), which is a method for optimizing a useful class of mathematical expressions under various constraints extremely efficiently. You can, for example, solve scheduling, assignment, transportation, factory layout, and similar problems with millions of variables in seconds. These techniques came out of large-scale government and especially military logistics and decision-making needs of the mid-20th century, and have now been applied extensively in many industries. Have you seen the UPS “We (heart) Logistics” ad? That’s OR.

"Data Scientist" and other titles

harlan.harris@gmail.com (Harlan Harris) — Sun, 13 Feb 2011 00:00:00 +0000

Neil Saunders has an interesting (to me) blog post up this morning, with the title “Dumped on by data scientists.” He uses the use of “data scientist” in a Chronicle of Higher Ed article to rant a little bit about the term. For Neil, it’s redundant, as the act of doing science necessarily requires data; it’s insulting, as if “scientist” wasn’t cool enough and you have to add “data”; and it’s misleading, as many people who call themselves “data scientists” are actually dealing with business data rather than scientific data.

Data-Science on Harlan D. Harris

You Don't Need a Search Engineer; You Need a Search Scientist

A Collaborative Template for A/B Tests

The Five Types of A/B Test Decisions

p Values Are Useful for A/B Tests, Sometimes

Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

Staff Data Scientist: Comments on Will Larson's Staff Engineer Book

A/B Testing and Product Ratings, Part 1: Delays and Bias

New Publications and Upcoming Talks

A few things I learned from two small Data Science conferences

What do Data Scientists mean by “Scaling”?

Conference Blogging INFORMS Analytics 2017

Transparency, Trust, and Proprietary Predictive Analytics

Insights from a Predictive Model Pipeline Abstraction

Patterns for Connecting Predictive Models to Software Products

Thoughts on Managing Data Science Team Workstreams (and a Shiny app)

Building a Complementary Data Science Team

INFORMS Business Analytics 2014 Blog Posts

Operations Research, from the point of view of Data Science

What is a “Data Product”?

Why a Data Community is Like a Music Scene -- Resources

More posts on the Data Community DC blog

Communication and the Data Scientist

integrating R with other systems

Survey of Data Science / Analytics / Big Data / Applied Stats / Machine Learning etc. Practitioners

Data Science, Moore's Law, and Moneyball

On "Analytics" and related fields

"Data Scientist" and other titles