The Daily Parker

There is a non-zero chance that my household will expand by 100% this week. Stay tuned.

Has this really been a full year? March 11th and 12th seem to be the days when everyone realized this was not a drill. John Scalzi:

I was on the JoCo Cruise at the time and had intentionally avoided news up to that point, but then two things happened. One, people came up to me wanting to tell me about Tom Hanks contracting the COVID virus (people knew that I know him personally), and two, my editor Patrick sent me a cryptic email telling me that I should call him immediately. After reminding him I was on a cruise and the ocean does not have cell phone towers, he told me via email that my book tour was cancelled and that plague was everywhere. I gave in at that point and caught up with the news from the world, all bad.

Josh Marshall:

As evidenced by what I tweeted on March 10, 2020, our last day in the New York team’s Manhattan office, I and a lot of my colleagues didn’t expect this to last a full year and longer. We sipped some whiskey as we locked things down for what we expected to be a month or two away. We were naive to the severity. We didn’t expect the catastrophic loss or the debilitating fear or the deep ineptitude of the previous administration’s handling of the virus.

And me:

What an exciting 24 hours.

President Trump made a statement from the Oval Office last night about the COVID-19 pandemic that completely failed to reassure anyone, in part because it contained numerous errors and misstatements. By announcing a ban on travel from the Schengen area of 26 European countries that applies to non-US residents, he enraged our European allies while doing nothing to stop the spread of the virus for the simple reason that the virus has already spread to the US. Not to mention, having a US passport doesn't magically confer immunity on people.

Meanwhile, historian John Barry, who has written a book about the 1918 influenza pandemic, points out the grave dangers in giving up masks right now:

There is no reason to expect that this virus will suddenly turn into 1918. There are limits as to how far it can mutate. But the more people who abandon masks and social distancing, the more infections can be expected — and the more variants will emerge.

In gambling terms: If you roll the dice once, yes, there is only a 2.77 percent chance you will hit snake eyes. But if you roll the dice 100,000 times, it is virtually certain snake eyes will come up several thousand times.

We know masks decrease transmission. Lifting a masking order not only means more people will get sick and die. It also gives the virus more rolls of the dice. That is a fact.

We're close to the end of this tunnel. But what a long year we've had.

Three reports of deaths today, two of them institutional. First, the one with the most relevance to me personally, one of the people most responsible for my sense of humor, died yesterday at 91:

Norton Juster, the celebrated children’s author who has died at 91, stumbled into literature much as his most famous hero, Milo, stumbles into the marvelous world of wordplay and ad­ven­ture in the classic 1961 volume “The Phantom Tollbooth.” They were bored and entirely unsuspecting of the wonders that awaited them.

A budding architect with a self-confessed tendency to procrastinate, Mr. Juster was living in New York City and working — or not working — on a children’s book about cities. A Ford Foundation grant had given the project a degree of urgency. But it was not the book he wanted to write, and soon enough, he recalled years later, he was “waist-deep in stacks of 3-by-5 note cards, exhausted and dispirited.”

To pass the time, Mr. Juster began scribbling the story of Milo, a boy of about 9 or 10 years with no interest in the tedium of school or “learning to solve useless problems, or subtracting turnips from turnips, or knowing where Ethiopia is or how to spell February,” and who was as bewildered by the grown-up world as grown-ups were by him.

Only slightly less relevant to me, writer Jelani Cobb joins the crowd of people observing the death of the Republican Party:

The most widely debated political question of the moment is: What is happening to the Republicans? One answer is that the Party’s predicament might fairly be called the revenge of “the kooks.” In just four years, the G.O.P., a powerful, hundred-and-sixty-seven-year-old institution, has become the party of Donald Trump. He began his 2016 campaign by issuing racist and misogynistic salvos, and during his Presidency he gave cover to white supremacists, reactionary militia groups, and QAnon followers. Trump’s seizure of the Party’s leadership seemed a stunning achievement at first, but with time it seems more reasonable to ponder how he could possibly have failed. There were many preëxisting conditions, and Trump took advantage of them. The combination of a base stoked by a sensationalist right-wing media and the emergence of kook-adjacent figures in the so-called Gingrich Revolution, of 1994, and the Tea Party, have redefined the Party’s temper and its ideological boundaries. It is worth remembering that the first candidate to defeat Trump in a Republican primary in 2016 was Ted Cruz, who, by 2020, had long set aside his reservations about Trump, and was implicated in spurring the mob that attacked the Capitol.

One of the most telling developments of the 2020 contest was rarely discussed: in August, the Republican National Convention convened without presenting a new Party platform. The Convention was centered almost solely on Trump; the events, all of which took place at the White House, validated an increasing suspicion that Trump himself was the Republican platform. Practically speaking, the refusal to articulate concrete positions spared the Party the embarrassment of watching the President contradict them. In 2016, religious conservatives succeeded in getting an anti-pornography plank into the platform, only to be confronted by news of Trump’s extramarital affair with the adult-film performer Stormy Daniels. Now there would be no distinction between the Republican Party and the mendacity, bigotry, belligerence, misogyny, and narcissism of its singular representative.

In addition, the G.O.P.’s steady drift toward the right, from conservative to reactionary politics; its dependence on older, white voters; its reliance on right-wing media; its support for tax cuts for the wealthiest Americans; and its increasing disdain for democratic institutions and norms all portend increasing division and a diminishing pool of voters.

Large things die slowly, though, and the GOP's rot and sickness will pollute our civil discourse for years to come.

Another large, old thing that may not die for a couple more centuries has also let out a gangrenous burp this past week, when Harry Windsor and his wife Meghan unloaded on "the Firm" they recently quit. Leave it to an Irish writer (in this case Patrick Freyne) to sound exactly the right note:

Having a monarchy next door is a little like having a neighbour who’s really into clowns and has daubed their house with clown murals, displays clown dolls in each window and has an insatiable desire to hear about and discuss clown-related news stories. More specifically, for the Irish, it’s like having a neighbour who’s really into clowns and, also, your grandfather was murdered by a clown.

The most recent internecine struggle is between the royal family and a newly disentangled Prince Harry and his wife, the former actor Meghan Markle. Traditionally, us peasants would be nervously picking a side and retrieving our pikes from the thatch. Luckily, these days the pitched battles happen in television interviews.

Over the course of the interview Harry and Meghan, who are charming, clever and good at being celebrities, make the monarchy look like an archaic and endemically racist institution that has no place in the modern world. Well duh. And despite all the outrage you might read in the UK tabloids right now, they also did something else that renders everything else irrelevant: they officially launched themselves in the United States.

Harry and Meghan are ultimately going to win. Despite the tabloid frenzy, this was never the story of an ungrateful pauper being elevated by the monarchy. This was about the potential union of two great houses, the Windsors and Californian Celebrity. Only one of those things has a future, and it’s the one with the Netflix deal.

Well, OK, only one of those things has actually, permanently died. But I expect the other two will die before I do.

I'm back in my downtown Chicago office after working at home every day for almost exactly four months. It's weird, as I'm once again the only one on the floor, but that will change pretty soon. And I'll still be working from home three days a week.

I did miss the view.

I've already done 8 km of walks this morning, and tomorrow I'm doing another 9. (Tomorrow's will end at Sketchbook Brewing, so I'll be even more motivated.) After being cooped up at home and forced to get my daily steps bundled up like the Michelin Man for a few weeks, I feel a bit liberated. The sidewalks are almost all clear (except for a few buildings whose owners suck, like the Cagan Management-run apartments near me), it's already 8°C outside, and the sky is crystal-clear. Tomorrow we might get a little rain before 9am but the afternoon looks absolutely gorgeous.

Spring hasn't officially begun yet, but it sure feels like it.

Microsoft has started sending little reminders of things that happened "on this day," no doubt taking cues from Google Timeline and Facebook Memories. But I did enjoy getting a reminder that I took this photo 14 years ago this morning:

Parker at Bardwell Park, Evanston, Ill., 18 February 2007.

It'll be 3 months tomorrow. I do miss him.

This morning I posted about some frustrations in getting our CRM system to import donations from our fundraising events so that we can then match donations with addresses to send out end-of-year tax letters. The frustrations have grown to the point where naming names seems appropriate, if only because Neon One, the CRM company, has a web-based ticketing system that doesn't really handle the level of detail their developers will need to (a) understand the problem, (b) understand the frustration, and (c) understand the features needed to solve (a) and (b).

As you read this, keep in the back of your head that I'm a software developer with 25 years of professional experience and another 15 of hobby experience before that. In other words, I've been writing software longer than almost everyone at the CRM developer has been alive.

Neon will probably consider this a feature request, though on any of the product teams I've run in the past 15 years, this would be a usability bug report.

tl;dr: Neon's import feature is software-centric, not user-centric. Instead of the feature helping the user, it expects the user to help the software. This causes grief for any user who is not a piece of software.

The simple problem

Because Neon doesn't have adequate (or, it seems, any) support for silent auctions and other day-of-event realities, we use a different system for our fundraising events. The other system spits out perfectly reasonable Excel documents with all the information we need to track donations along with the fair-market valuations of silent auction winnings. We want to import that information into Neon so that we (a) can print out end-of-year tax letters and (b) accurately track giving in the long term.

The obvious path, which doesn't work

People have studied usability for almost as long as I've worked in software professionally. Jakob Nielsen has written about since 1998. The first principle of usability has never changed: make the obvious path work in an obvious way.

Neon has an import function that appears, at first glance, to import exactly the kind of data our event system produces. It seems like one should be able to import a flat file containing the donor name, address, email, and phone number; the date and amount of the donation; maybe a note or other optional information. You would think you could map the columns on the file to fields in Neon, and the software would read the data file and import the data. Maybe you'll get one or two spurious donor records when the information in the file doesn't exactly match an existing record's data, so maybe you'll have a few minutes of clean-up that you can do right from the import report after it's done.

Anyway, that's how I'd design it. That's how Jakob Nielsen would design it. That's how the 22-year-old newbie with a still-wet bachelor's degree in design would do it.

That's not how Neon designed it.

OK, so there may be an extra step

The first thing the Neon import feature asks is: does your data have Neon account ID numbers? If not, it warns you it won't be able to match your data with existing donor records.

Wait, what? The CRM already has a decent "find duplicate records" feature, so why can't this run automatically on new or imported entries? (Seriously, Neon: why do we spend time after every concert de-duping our data because your software doesn't think to match existing records with new ticket orders even when all of the data are the same in both records? More on this in a moment.)

All right. I'll spend 15 minutes adding Neon IDs to all the donation records in the exported file that correspond to existing donor records. It's an extra step that the software should be able to accomplish on its own, but whatever, no one likes writing record-matching code.

Now what I expect is that Neon will add the new donations to the existing records, and add new account records if I don't supply an ID.

Nope. My first pass through this process looks only at the exported records whose IDs I've provided and updates those donor records, while completely ignoring the new records. And it then takes me to a screen that looks suspiciously like it will make a total hash of the donation records in the same export file if I click "next." So I abort.

Maybe a few extra steps?

I think, perhaps I should give Neon a little extra help now. Let me scrub the data going into the import so that we have the best chance of good data getting into the CRM. So I separate the export into two files, one containing Neon IDs and one with all the new donors who came to the event. Then I go through both to make sure mailing addresses conform to USPS standards, phone numbers are uniformly 10 digits without separators, and email addresses are validly formed.

Next, I import the file that does not have Neon IDs, hoping it will create new records for me. It does! And it even exports a report containing the new IDs it created, albeit with only the donor's full name and not the donor's first and last names, which creates a bit of extra work as now I have to manually map them to my donor export.

So after I add the new IDs where needed in my donation export file, I'm ready to import the donations. I start the donation wizard, it accepts that my records have valid Neon IDs, and I map the donation date and amount columns to Neon fields. And then it tells me that I don't have a "donation type" mapped.

I'm not going to go through the steps required to figure out what a valid "donation type" is. Suffice to say, I add a column to the export called "Donation Type" and fill every row with the value "Donation". (Why can't we just specify constant values for required fields at import?)

I go through the import wizard again, map everything required, and hit import. Oof! Error! Apparently, dates are hard. The export file has donation dates in the format "Jul 17, 2020 7:27:19 PM", but Neon says "Time field must be a date;now it supports 'MM/DD/YYYY','MM-DD-YYYY' and 'MMDDYYYY' format."

Now, without going into the rabbit hole too deeply, a few things immediately occur to me as someone who has successfully parsed dates for a quarter century in a half-dozen programming languages. First, why the fuck does Neon only accept those three formats when the format presented is unambiguous and can be parsed by nearly every programming language out there? Second, none of the formats presented or accepted in this case conform to ISO-8601, the international standard for date and time representation, so everyone is on shaky ground here. Third, I got all the way to this point and now it tells me I have to go all the way back and change the date format by hand? Because it turns out, Excel can't parse the dates either. Good work, 3rd-party event package. Nicely done.

Once more unto the breach, dear friends, once more;
Or jam the import with our ISO dates.

After entering all the dates by hand (made easier by 90% of them being the date of the event), I re-exported the file to .csv and tried the import again. (Oh, yes, Neon can't read XLSX files, so I have to export to CSV every. Single. Time.)

Success! Finally!

And now I get to do it again with the silent-auction winners list. All of it. Again. This time I just entered the 9 new accounts by hand, and discovered 7 duplicate accounts that had to be merged, and thanked the universe we only had 60 silent-auction items.

So I go to import and...Goddammit I forgot the "donation type" field.

So I go to import again. And it mostly worked. Except even though I specified which field contained the fair-market value to map against the donation, Neon ignored that. It appears nowhere in the individual donation records. Why even present the field as an option if...I mean...what the hell, Neon? Yet more shit I have to map by hand later on.

But wait, it still doesn't work

After all that effort, I did some spot-checks on various accounts and found that even though none of the donation records shows fair-market value for auction items, at least all of the donations that I expect to see in each record appear in each record. So now it's time for the 2020 donor report, and...

Um...you're fucking kidding me.

None of the imported donations shows up in the donor report's "2020 Donation Amount" field. After checking a few donor records, I promote my hypothesis (that Neon groups by the record-creation date for the annual sum instead of the actual donation date) to a theory. Neon support case #00316491 is born.

How Neon should have coded this

Many, many developers have solved this problem before. Importing donations should require only one pass through the Import feature, and follow this heuristic:

1. As soon as the user selects "import donations" from whatever UI control offers the choice, Neon presents a page of simple documentation explaining the process, listing the required fields, and offering some insight into how it will work.
2. The user selects the import file, which can be CSV, Excel, or any other common delimited format.
3. The user maps all of the relevant columns from the import file to Neon fields. Neon does not present the user with fields that it will ignore for no stated reason later on.
4. The user provides constant values (from drop-downs if necessary) for required fields that do not appear in the import file.
5. The user clicks "upload."
6. For each record:
   1. If the imported donor account has an ID that matches an existing account by ID, use that donor account.
   2. If the imported donor account matches the name and email of an existing account (or some other criteria), use that donor account but flag the row for review.
   3. If the imported donor account does not match an existing account on either criteria, create a new donor account.
7. Stage the donation record with the specified, found, or new account ID, including all of the fields that the user mapped in step 3.
8. Present the list of "for review" items to the user before committing the import, allowing the user to make edits to the imported data, or move back a step in the process.
9. Once the user is satisfied, the user clicks "commit" to write the data to Neon.

I get it: Neon's dev group have process problems

I mean, guys, this really isn't that hard. You want hard? Write an IBM360 to MS-DOS import, complete with different endian values, and where mapping has to be hard-coded because configuration files haven't been invented yet.

I know how software development works. I expect any Neon folks reading this may think this is unfair, that the devs told management the features weren't finished, that management told leadership the devs weren't 100% finished but the stuff works well enough, and that leadership looked at the growing list of must-have features for the brochure and forgot to finish the must-have features for the users, that "it's not my fault." I'd also bet you a dollar that any dev reading this will think "I told you so" (unless they think "it works on my machine, you DFU," in which case you have other problems.)

In other words, you guys have a process problem. Somewhere the definition of "done" that passed QA for the import features didn't match the definition of "done" that users need.

For this we're paying $3600 a year. NB: we're willing to pay a lot more for software that works the way we need it to.

Meanwhile, though, I'll have to hand-correct what all this automation should have given me already, and get the damn tax letters out.

And Neon CRM support incident #00316497 is born.

I'm president of the Apollo Chorus of Chicago. One of my jobs is to send out letters to all of our donors acknowledging their donations for the previous calendar year. These letters should have gone out by January 31st, but...well...OK, I'm a little delinquent. And for no other reason than I really, really did not want to merge all the data by hand.

You see, we use a smallish CRM system for all of our institutional data, which works pretty well, especially with our membership and tech-savvy donors. Many people have set up recurring donations through the CRM portal (please donate!), which happily bills their credit cards each month and creates new records for us to merge later. (It's really, really dumb about preventing duplicate records, but its merge feature works well enough.) Other donors make ad hoc contributions through the CRM as well. The CRM then sends a report to our treasurer which he imports into QuickBooks, and all is good.

You know there's a "but." See, we use a different system for managing our annual fundraiser, because our CRM sucks at event management. This system records all of the donations received for each event, plus silent auction winnings, and produces its own reports that we import into QuickBooks.

The upshot is that the CRM is not the single point of truth for donations, though it muddles through as the single point of truth for membership and music purchases. The Development Committee therefore doesn't want to use the CRM, even though it's why we have the CRM in the first place. The treasurer doesn't want to enter or reconcile all the event donations by hand either. Nor does our IT Director want to merge all the duplicate records that would result from importing the event data.

I hope to have a solution for this by next year. This year, however, I'm banging my head onto my desk as I try to reconcile QuickBooks' list against the event software's list against what should be the single point of truth for all of it.

Facebook reminded me this morning that 10 years ago today I got the first digital camera I've ever owned whose photo quality approached that of the film cameras I had growing up. My new Canon 7D replaced my 5-year-old Canon 20D, and between the two I took over 32,700 photos in just over nine years. In May 2015 I upgraded to the Canon 7D mark II, the first digital camera I've owned whose capabilities exceeded my 1980s and 1990s film cameras.

I've updated the chart showing all the photo-capable devices I've owned since I got my first SLR in June 1983, along with other data showing, to some extent, how technology marches on:

Here are four example photos. (To see all the details, right-click the photos and open them in separate windows.) First, one of the earliest photos I took with my AE-1 Program, in Raton Pass, N.M., mid-August 1983:

Keep in mind, this is a Kodachrome 64 photo scanned some 38 years later, with a bit of help from Adobe Lightroom. Printing directly from the slide would make a better-looking photo...maybe. In any event, the resolution of the slide exceeds the resolution of the scan by an order of magnitude at least, so there really is no way without specialized equipment to produce a JPEG image that looks as good as the slide itself.

Jump ahead a few decades. Here's an early photo I took with the 20D on 20 May 2006 in Portsmouth, N.H.:

The original photo and this edit have the same resolution (2544 x 1696) and the same format (JPEG). Other than a few minor burns and dodges, this is what the camera recorded. It almost approaches film quality, but had I shot this image with Kodachrome 64, it would have much more vibrant color and a depth of texture that the 20D just couldn't achieve.

Now from the 7D that I got 10 years ago today, near Saganonomiyacho, in Kyoto, Japan, in November 2011:

With the first 7D, I gave it a 32 GB memory card and switched to the lossless CR2 format. The JPEG above has as much depth and range as a JPEG can have, but the CR2 file it came from finally has as much detail and photographic information as consumer-quality negative film from the 1980s or 1990s. Its 5184 x 3456 resolution comes awfully close to the density of, say, 100-speed Kodacolor VR-G from the mid-1980s, but the 7D's CMOS chip has literally 32 times the sensitivity of the fastest consumer film then available (ISO 12,800 vs. ISO 1600). I shot the photo above with a focal length of 250 mm from a bridge 250 meters from the subject, at ISO 1600, 1/500 second at f/5.6. The same shot on Kodacolor VR-G 1600 would have massive grain, and the same shot on Kodachrome 64 would have required an exposure of 1/15 second—guaranteeing camera shake. (I've re-edited this photo slightly from the quick-and-dirty treatment I gave it in my Tokyo hotel room after getting back from Kyoto.)

Finally, take a look at this photo from my current camera, the 7D Mark II, of the Chiesa de San Giorgio Maggiore, Venice:

I mean...wow. Even cropped slightly from the raw photo's 5472 x 3648 resolution, the detail is just as fine as Kodachrome 64 ever gave me. I shot this at 1/1000 sec., f/5.6, at 105 mm using a borrowed EF24-105 f/4L lens. (I posted a similar shot in June 2015 when I got back from the trip. I think this one has better composition and editing.)

One more thing, which I won't illustrate with a comparison photo but which I do think bears mentioning: these days, I only pull out the 7Dii for serious work. For day-to-day photos and snapshots, my smartphone's camera works better than digital SLRs from 15 years ago. We do live in the future.

I'm once again not going to get to 10,000 steps today, and that bothers me irrationally. I just need to accept that when it's -10°C and snowing, and I've got a full day of work and chorus business to do that will take me until 10pm, it's OK not to take an hour-long walk to get the steps I need. I'll still manage over 5,000, and I'm certain I'll hit 280,000 for the month. (The worst month I have on record was January 2015, when I got 310,514 steps—just barely 10,000 a day.)

OK, I'm now going to bundle up, put on boots and a mask, and go for what will certainly be a brisk walk. And then I'm not going outside again until tomorrow.