Thoughts On Multi-Site QA: Best Practices & Pet Peeves

 There are 2 primary audiences for QA. The first is local site implementers (SDMs) who want confirmation/proof that their implementations pass at least minimal muster, or else know exactly where their implementation has gone wrong.  The second audience is potential users of the data, who are similarly looking for assurance that the data at the sites will meet minimal expectations and generally be usable for a given study.  In addition, the second audience will often want to see basic descriptives on the various fields, in order to see e.g., just how much gender identity data you have.  These statistics are also good for forming opinions on how homogenous/harmonious the implementations seem to be across sites. If one site shows an average of 7 ambulatory visits per member per month, and everybody else is around .4, then that’s maybe worth looking into.

Generally, you will want to create 2 separate reports in a QA process, one for each audience.

The presumption is that QA failures will either be fixed or documented as not fixable. So new fails should be actionable by the implementing site (unless they are known issues that have been explained on the issue tracker. If an issue isn’t worth following up (e.g., 3 stray records out of 8 million) don’t put a fail in the report.

Best Practices/Pet Peeves

1. QA Programs go hand-in-hand with the specs to define a given data area. Expect to hear from SDMs running your program about differences of opinion/interpretation of the specs. It may take some dialogue & iteration to settle on checks and tolerances. You should list the objective checks & those tolerances on your collated report along with (at least) a summary of the results of those checks at the sites.
2. Don't make me hunt for the table listing checks & pass/fail results--that should be the first thing I see, right at the top of the output.
3. I should be able to ctrl-f for the word 'fail' and instantly know if I have any fails & if so what they are.
4. Be thoughtful about what reporting output you create for the local implementer/user. In particular, thou shalt not spit proc contents output into a report.  If you want a full CONTENTS on my dset, run that out to an output dataset and consume that in your collated report. Don’t forget that you can produced most anything you want once you get results back from the sites.
5. Make your collation process and collated report creation process as streamlined as possible, and favor publishing the results immediately after you get updated results.
1. You should be able to receive updated results at any time, and have only to unzip contents into a folder, run a single program, and then upload the resulting collated report.
2. Give Sites immediate credit for having fixed fails
3. While it would be wonderful to always have the time to pore over results and compare site implementations & follow-up with individual sites, it is also not crazy to basically crowd-source this work.
6. Produce the smallest number of output datasets as will hold the data you need.  Each one has to be checked for PHI before it can be returned.  Consider concatenating multiple similar datasets together.
7. Save off close-to-raw records that violate a check and point the user to that dset in the part of the report that lists the fail, so they don't have to write their own code to (dis)confirm the issue & see some samples data to start figuring out how it got into the dset.
8. Graphics are way better than tables.
9. Format values to make them interpretable to ppl who have not memorized the spec.