You're telling me what you need--I'm telling you what we have.

Listening in on some of the PCORI Workshop To Advance Use of Electronic Data, I can't help but be reminded of this bit of the movie Apollo 13:

For organizations like the HMORN, it's crucial to remember that we are data recyclers--we don't get to say what the health plans/providers record in the conduct of their business. We get what we get.

A big part of the valuable expertise that we've built up over the years is how to wring the greatest amount of information possible out of the data that we do have.

A Design Pattern for Production SAS jobs

Okay, that title is probably putting on airs.  But I have been thinking about what a good set of conventions might be for us to use at GH, and I think I've happened upon one, and wanted to flesh it out a bit to see if it looked reasonable all in one place.

So here it is.

Disclaimer

Lots of this relies on my personal preferences (e.g., many small files vs. one single monolithic job) and what may be unusual bits of the GH (or my personal) computing environment.  I do not intend to prescribe these as The One True Way for any and all people who may eventually read this.  I'll try to state my reasons for preferring something when I have reasons.  If I don't state any, feel free to assume that I don't have any & it's just arbitrary preference (or ask in the comments & I'll respond).

Some Context

Couple of salient details about the environment I'm working in.

• (As of this writing anyway) the primary destination for the data marshaled in these jobs is SAS datasets sitting on a central "data warehouse" server.  That's a windows server & is  a SAS/CONNECT rsubmit target.
• Source data come from any or all of:
  • SAS datasets sitting on the same destination server.
  • SAS datasets sitting on a unix server (which is also a SAS/CONNECT rsubmit host).
  • GH's Enterprise Data Warehouse--an RDBMS.
  • GH's Cost Management Data system--a different RDBMS.
  • Clarity--on still a third type of RDBMS.
• My primary machine is a laptop, which often has only a poor connection to the network.
• There are 3 of us working on "data infrastructure".
• We use git for source code control. 

Goals

We want:

• all jobs to run on a schedule, so the humans don't have to remember to kick things off.
• to be able to roll back to a previous version of a file quickly in case there are errors or bad updates.
• to be able to make and test fixes to jobs quickly, and without disturbing the user-facing (production) data.
• it to be reasonably easy for a new person to understand/maintain a job they didn't create.

That last guy is the reason for the conventions.

The Conventions 

Destination Data

Folders

Datasets should live in folders that group them by content area.  (Right now we have all VDW datasets in a single folder, but the idea of putting each one in its own subfolder sometimes appeals to me.)  So encounters/pxs/dxs for instance should all be in a single folder.

• Current production versions of each set of datasets should live in a top level folder, whose name signifies what datasets should be found therein (e.g., "ambulatory_visits").
• Only production data (and associated views thereon) should live in these folders--there should be no ancillary or interim datasets, documents, programs, or other files in there.
• Any ancillary and end-user-useful interim files should live in a subfolder called support.
• Prior versions of datasets should live in a subfolder called past.  Space permitting, the immediate prior version of the dataset should be stored unzipped.
• New versions of datasets should be stored in a subfolder called next, for whatever duration of time it takes for staff and/or users to validate them. 

So jobs build datasets in the /next subfolder, whatever QA & or user validation is going to happen happens, and then dsets migrate from prod -> past, and from next -> prod.  This helps cut down on runtime errors due to users holding locks 

Datasets

• Dataset names should clearly convey what data is contained therein.
• Datasets should be labeled to indicate their contents and their expected-complete-through date (e.g., "Professional claims paid through 31oct2012").
• Variables should similarly be labeled descriptively, and include important caveats if necessary and/or aids to interpretation (e.g., 02 = breast, 03 = colorectal).

Source Code

Control 

In case it doesn't go without saying, all source code needed to produce production data, as well as any associated notes, documents, ancillary data, etc. should be under source code control.

The 'production' repository is the one named in the scheduled job.  At GH we store source in a separate folder structure that largely mirrors the datasets that the jobs create..

Substantive edits to source code must not be made directly to files in the production repo--you should always do real work on a clone whose changes you then pull into prod once they are tested and ready.  The only edits you should do in prod is the minor alterations necessary to have the job run in production mode (that is, so it runs successfully under a service account & replaces the prod data).  Unless you're feeling lucky. ;-)

In my case I usually have 3 repos/job: production, a working copy that lives on my laptop, and a 'staging' copy that lives on the network in a non-production area.  I typically edit on my laptop, pull changes into staging, run/test them from there (usually from an ad-hoc task scheduled on a machine that stays connected to the network).  I use this script for that scheduling, so I get an e-mail when the job finishes.  I do this because the jobs often run longer than I'm connected to the network and I want to run them without interruption.  So the sequence is usually:

1. Clone prod onto my laptop.  Work there.
2. Clone laptop into staging.  Test there.
3. Pull from staging into prod.
4. Pull from prod into laptop to bring that repo up-to-date.

Round-and-round she goes.  Git makes this muy easy.

Organization

Here's where the real fun is.

Like datasets, every job gets its own folder.  The programs necessary for the job all live in the top-level.  I have the following subfolders:

• /tests: for holding unit test type programs--optimally one/macro in the job.
• /supporting files: reference data (usually mdbs, for holding things like our native lab result codes & the VDW test_types they should be translated into), reconnaissance programs, ad-hoc fix programs; sketches for new approaches to things
• /qa: data QA
• /output: QA programs spit results out here.
• /notes: various text files, word docs, etc. describing the job, or questions I asked/answered in the course of the programming.

Also like with datasets, the top-level folder has nothing but the .sas files necessary for the job.  (Because this stuff is all under SCC, we don't have to be sentimental/anal-retentive about obsolete code--we can go back in time anytime we want.)

There are two programs that are part of every job:

1. main.sas, which  is the entry-point for the job--that's what controls execution.
2. environment.sas, which--wait for it--sets up the environment for the job, making it possible/easy to switch the job from "dev" mode (runs as me; maybe on subset data; does not disturb production data) to "prod" mode (runs under a service account; puts data in public-facing places).

By 'environment', I mean things like:

• Where the job will write out any interim datasets.
• Where it looks for ancillary datasets (e.g., our enrollment job has a separate dset of DRUGCOV information that lives in a file that only gets built once in a while--that type of thing).
• Where it looks for files of account credentials/rlogin script files (these are different if I'm running in dev mode (runs as me) vs. production mode (runs as a service account). 
• Crucially--where the job will write out its final datasets.
• Whether are any global subsetting parameters or "obs = X" type conditions that should be in effect during the job.
• Sometimes--the locations of the source data for the job.

Usually I set all these various libs/parameters up to prod-mode values, and then define a macro that sets them to dev values.

I would like to say that environment.sas is the only place you'll find a libname statement in my jobs.  I aspire to that, but it's sometimes tough when I have to rsubmit into server1 to do some stuff, and then rsubmit from that remote session into server2 to do some other stuff.  I'd love to hear ideas for overcoming that.

In addition to these two programs, there are a series of files that define the macros that actually get the work done.  There's typically nothing fancy about these macros--they just take a few parameters (usually naming datasets they should accept as input and spit out as output) and do a chunk of work, assuming that their environment is set up for them.  The macro-fication is strictly about being able to control which bits of the program run when.

So main.sas typically looks something like this:

** Start a remote session on the server. ; 

%include "//server/share/start_rsubmit_session.sas" ;

** Make sure only one of these is in effect (or that the last definition is ;

** the one you want). ;
%**let codehome = //server/share/job_name/ ;       ** <-- prod. ;
%**let codehome = //other_server/share/job_name/ ; ** <-- staging ;
%let codehome = //roys_laptop/share/job_name/ ;    ** <-- laptop ;

%include "&codehome.environment.sas" ; 

** Put us into development mode (comment this out to go prod) ;
%set_dev ;

%include "&codehome.gather_mbhist.sas" ; 

%include "&codehome.gather_retro.sas" ;

%include "&codehome.interleave_enrollment.sas" ;

%gather_mbhist(outset = mbh, obslim = &obs) ;

%gather_retro(outset =  retr, obslim = &obs) ;

%interleave(mbhist = mbh, retro = retr, obslim = &obs, outset = retro_enrollment) ; 

I've seen jobs where the files are named with leading numbers to indicate the order in which they are run.  That can be nice, but I don't tend to follow that convention myself.

There are a couple of advantages to setting up jobs this way:

• It is very easy to see what mode a job is currently in, and to change it to a different mode.
• Because main.sas is the only program actually executing code, I can use it to easily control the job.  If something barfs in a middle step I can comment-out the bits that ran properly and manually submit the rest.
• If I'm smart about how I name the macros that make up a job, main.sas can read like a description of the high-level steps of a job.
• Having code split up into discrete chunks makes it easier to test.  I can have an e.g., job_name/tests/test_gather_mbhist.sas program that just does minimal environment setup & then %includes/calls that one bit of the job. 
• Putting the macros in different files lets me navigate to the code of a given step more easily than if I just had a single monolithic file with everything in it (I use an editor w/a multi-tab interface, so it's easy to jump around). 

So there it is.  What do you think?

Lives Schmives

We and other networks like to toss around "covered lives" as a statistic with which to impress funders. I think this probably claims too much and too little at the same time.

It's too much because we don't literally have e.g., 13-million complete birth-to-death life stories in our data. But it's too little in that looking at a snapshot of who's enrolled over any particular period of time understates the *depth* of coverage over time. A lot of our members stay members over long periods of time. Not all, to be sure, but plenty do. They retire; change employers; move from place to place, but they very often stay with our orgs.

So what would be a better statistic for us to brag on? Summing up person-years goes some distance to address the issue, though that too undersells the longevity. Here are some candidate statistics we might use to give a better sense of what we've got:

• total person/years
• the median length of terminated enrollments (or maybe show percentiles on that distribution)
• the proportion of terminations that are due to death.
• Average proportion of live covered (so--(#covered days /(DOB - min(DOD, &sysdate)))

What other statistics can you think of?  Leave a comment & make your case!

The difference between what we do, and Informatics

Okay brace yourselves, because this is brilliant. ;-)

There are 2 difference between us and informaticists.

First: we have more data than time, and they have more time than data. So we do "the simplest thing that could possibly work" to move data around & get substantive research done, and wind up with not much to brag about in terms of technical architecture. Informaticists spend their time bulding elegant architecture (which you need in order to impress your fellows & get published) which may not be practical for actual industry use.

Second, because we are largely data-recyclers rather than primary generators of data, we are less concerned with standards (other than healthcare and insurance industry standards of course) because it's not up to us what data we get. We get what we get, and make the best of it.

But alas, our funding is hostage to informaticists, and they require that we worship their gods...

VDW Q & A: CRN Scholars' Questions

In which I plan how to answer questions put to me by one of the moderators of the upcoming CRN Scholars concurrent session at this year's HMO Research Network conference...

Q: The VDW appears to offer the opportunity for new investigators to launch a multi-site study with relatively modest resources.  Is this true?

I would say it allows multi-site studies to spend much less on programming.  How much less is very dependent on the needs of the study.  If you can stay entirely within the domain of data that are available in the VDW, then you should not need very heavy programming work done at the non-lead sites.  So if in a pre-VDW world you might budget a biggish project w/30% FTE programmer at all of 6 sites, post-VDW you can probably get away with more like 40% at the lead site, and 10% or less at the other sites.  But that's a big 'if' of course--you can do a lot of studies just with VDW data, but you can't do everything.  And of course programming is just one of many items in a study budget--you've also got to have Investigators, Project Managers, and so forth.


Q: How much effort is required to obtain VDW data across multiple sites?

From a technical standpoint, not a ton, frankly.  You've got to write a VDW program of course, and writing those is generally more difficult than writing single-site extraction programs, just because you have to be extra defensive in your programming, and because there's usually an extra round of programs to write to collate data you get back from your sites.  But that work is amortized over your sites & so in the aggregate it's a win, effort-wise.

But of course that doesn't really touch much of the effort in doing multi-site research.  You've got to engage with scientific staff at the sites to get collaborators.  You've got to get a project funded, and teams at the sites staffed.  You've got to get someone to run that program you've written, and send you back the results.  You've got to have conference calls and agree on exactly what bits of data should move from one site to another (just aggregates?  person-level?  event-level?) and what by what means?  VDW doesn't solve any of that of course.  It's very very useful.  But not magic.

Q: What are the specific tasks that have the VDW Implementation Group, content area Work Groups, and Site Data Managers have done “up-front” to make it easier for researchers to obtain their research data, and to combine similar data across sites?

Of those three groups, the efforts of the Site Data Managers--the people actually implementing the VDW files at the sites--dwarfs that of the other two groups.  That's just inescapable--the rest of us can plan, scheme, pontificate even, but the only people that can actually touch/poke/prod data, and have any clue what's going on on the ground at site X are the people at site X.  That's just comes with having different sites.

Probably the biggest thing SDMs have done (at most sites anyway) is simply collating the data from X different disparate local systems into the very few files that comprise a VDW implementation.  So for instance at Group Health, in order to get utilization, px & dx data from 1993 up to the present, from inpatient & outpatient settings, we have to hit something like 13 different data systems (and we are not the most complex site, not by a long shot).  Some are claims-based, some are EMR-based, and some are pre-EMR legacy system data.  Having data from all of those systems in a single set of 3 files is enormously convenient, even just for single-site studies. (Which is why most VDW files are sources of first resort for all of our programmers--not just the ones on multi-site studies.)

So--SDMs good.  SDMs wonderful.  But the other groups also contribute of course.  The Work Groups steward our specifications, which are our precious islands of shared understanding, in a sea of ambiguity.  The specs are our touchpoints with one another--where we figure out just how alike and different our sites are from one another.  In addition to developing the specifications, they also construe them for implementers--by which I mean respond to questions, both abstract and concrete regarding implementations.

I've been doing multisite research programming long enough to remember the pre-VDW days, in which we would distribute English descriptions of the data we needed, have many many conference calls hashing out details of each site's implementations, and go back and forth over anomalies & misunderstandings.  And lots of that effort would be just plain duplicative--writing code to implement a particular definition of "continuously enrolled" for example.

And then at the end of the study you would throw it all away!  It was insane.
 WGs have also historically presided over our shared Quality Assurance efforts (and several have been going strong with QA work even in these intense days of V3 implementation).  That's an area that we hope to take up anew, now that we have the many many V3 changes (mostly) behind us.

As for the VDW Implementation Group & VDW Operations Comittees, I think of the former as the community of SDMs & the latter as those Scientist collaborators who have graciously chosen to help give us direction so the VDW stays relevant to the needs of HMORN researchers.  Strategizing for VDW is a very difficult thing, because the Scientist/users know the "bang" part of the equation, but the SDMs know the "buck" part.  So no single person can really balance everything.

Q: How consistent /uniform are the data across sites?

This is a horrible question to try and answer in the abstract.  The answer is of course, that "it depends".  Depends on which sites, over what time periods, what sorts of data, for what purpose, and what you mean by "consistent".

I will say that the domain of VDW applications for which you can use the data uncritically, without testing the waters, investigating whether the particular data you expect--those left-handed-Dentists-without-tonsils you love to study--are in fact there as you expect them to be, is close to empty.  Probably you can do quick counts for feasibility without investigating consistency/quality.  But short of that, users are well advised not to assume very much in terms of consistency.

The specs that we all code to mean the files should exist, the variables should exist, be of the type specified, and contain only allowable values.  We guarantee that the best humans we could find have populated those files at the sites, using their best judgment, the best data they have available, with the best guidance we could give them.  But our data are so thoroughly touched by fallible fallible human hands and minds (starting at the care setting mind you) that there are bound to be points of inconsistency--not all of which reflect e.g., expected practice variation.  VDW is not a magic bullet, alas.

This was revealed pretty starkly on the CRN's Pharmacovigilance project when we looked at chemotherapy data.  Chemotherapy can be coded in both pharmacy and procedures data, with varying levels of precision in each.  Across the eight sites on that study, we found that at some you'd find evidence of chemo almost exclusively in procedures data, and in others you'd find it only in pharmacy data.  This was due to differences in the way the different sites' health plans delivered and/or paid for chemo treatment.  Sites that didn't do any chemo in-house had everything coming off of rx claims data.

Q: The VDW tables are described as “Quality Assured”: What does this QA include?  What does it not include?

Frankly, I don't think I would describe them that way--not generally anyway.  I don't mean any disrespect to my colleagues on the Workgroups or at the sites.  I just don't think we have done enough with QA to really make a general representation that the data is "quality assured" (and what that means to different people will be different, so it's probably perilous to claim even under the best circumstances).  I know we at GHRI have certainly not done enough.  We aspire to it, and are striving toward better QA, but we are not even close to where we should be, alas.

So--that sounds really bad right?  Holy cow, this guy doesn't believe in his own project!  But here's what I will claim about VDW file quality: it is just about guaranteed to be better than what you would get if you put together ad-hoc extracts from indigenous sources for your study.  There are 2 reasons I'm comfortable claiming this:

1. we have better programmers at the sites than you're likely to get for a project, and
2. you're not the only one using the VDW--it has been heavily scrutinized by many prior uses.  That scrutiny discovers issues, which get fixed to benefit subsequent studies.

The first point owes to the generosity and foresightedness of the Site Directors, who agreed to get us the people we have.  The second is one of my favorite things about the VDW--when used properly, it is a vehicle for cumulating the value of programming efforts across many studies.

I could go on, but who will even read this far? ;-)

1970 called--they want their programming language back

I've had a couple of conversations over the past few months in which people have questioned whether SAS is really still our best bet for the VDW. That is a question worth asking--SAS is not cheap, and has some significant shortcomings. The amount of disk space (and I/O work) required for long text fields can be close to heartbreaking, for example. The SQL optimizer can be maddeningly stupid (and it's pretty easy to write programs that have no hope of taking advantage of indexes, even if you have a decent indexing strategy). There's no persistant data-serving process to cache data across requests--unless you have some smarts & memory devoted to disk access (e.g., SAN/NAS type stuff) each SAS job does its own I/O tabula rasa.

I'm writing to defend SAS as a data-access technology for the VDW. I believe it is the one bit of software that we should require sites to have licensed in order to play in our VDW reindeer games. The only bit.

Some Distinctions

To start out I need to make a couple of distinctions. The first distinction is user location--is the user for a given VDW query local (from my site) or remote (from a sister site)? The second distinction is between the VDW's user interface--the parts of the system that an end-user actually touches/interacts with (by, e.g., pointing/clicking around a graphical interface, or writing commands, or speaking into a futuristic microphone of some kind ("diabetics please")), and the parts of the system that actually store and enable access to the VDW data. Call this second distinction front-end vs. back-end. The front-end is what defines what an end-user can do with the data.

Right now our user interface is of course SAS. Anything you can express in SAS using the files/variables we have, you can do with the VDW.

Location, Location, Location

Local and remote users have very different abilities and expectations. Local users are bound to learn the local data environment (VDW and non-VDW alike) including any tools supported at the site for accessing their site's data. They can pretty easily eyeball the data, and see when something looks goofy. They will also have higher expectations around performance--they will be the ones actually staring at the hourglass or spinning beachball or "Query executing..." status bar message while their queries run. (They also know exactly who to complain to when these things take up too much of their time.)

By contrast, remote users are at two distinct disadvantages. To begin with, they have to elicit the help of a local user to actually cause their program to run. (This is by design--the VDW was/is fearsome enough to site staff, who so feared a loss of control over their data that it was difficult even communicating what the VDW is. We figured trying to automate the local humans out of the process would be a non-starter.) That involves e-mail and maybe phone calls. The local humans then need to make sure the query is approved (frequently by other local humans), run it, evaluate whether it ran as expected, report any errors, evaluate whether the output is within the scope of the approval, figure out a good means for sending the output (is e-mail sufficient, or do we have PHI or other sensitive data requiring fancier measures?) and then actually exercise those means. The remote user in this scenario should not expect results back in minutes, the way a local user would. Lucky remote users will get results back in a matter of hours or days. Unluckier ones, well...longer. Remote users know this, and tend not to spend too much time staring at their in-boxes waiting for results to come in.

They are also at a disadvantage with respect to learning the local data environment. They don't know what your servers are named, what sorts of pipes exist between where their program runs & where the data actually live. They don't know what operating system you're running, or what server database, or that your DBA has a weakness for chocolate and works from home on Fridays.

No doubt you can see where I'm going with these distinctions. Here's why I don't see VDW moving away from SAS anytime soon.

As a user-interface, it's incredibly expressive, and offers a lot of convenience tools that would be painful to reproduce or do without.

SAS has a decent SQL implementation, an excellent procedural language (data step) and even a capable macro facility (so you can write code that writes code).

Imagine having to implement the functionality of one or all of these in say, Java:

• TRANSPOSE
• TABULATE or REPORT
• UNIVARIATE

I'm no java programmer (my tastes run more to ruby and .net) but I bet you dollars to doughnuts it would be exceedingly unpleasant.

Even MEANS is a significant tool--try figuring out percentiles in say, SQL. No fun.

Even if SAS isn't entirely unique in these abilities (is SPSS a contender here maybe? R?) it's awfully good, even at large data volumes.

Just because SAS is the front-end does not mean you have to use it on the back-end

SAS has for years supported a suite of ACCESS products, which offer binary client access to such diverse products as:

• Oracle
• Teradata
• Netezza
• Sybase
• Informix
• MySQL
• OLEDB (which gets you MSSQL, and anything else anybody's written a driver for)
• ODBC (see OLEDB)

And (as they say) many more. If you license the right ACCESS module you can store your data pretty much anyplace you like.

Using SAS as the front-end means your remote users don't have to care what your back-end is

Because SAS/ACCESS pretty completely abstracts the back-end away, nobody remote really has to know or care what your back-end is, so long as it's available and has the expected data. It looks like a SAS dataset to SAS--I can feed it to TABULATE. Unless performance is so horrible that run time becomes a significant contributor to my wait (and recall all the waiting I'm having to do on the humans in the system) however many squirrels have to run on their exercise wheels to get me the data is really not my concern.

Now, will remote users get all the benefits of using e.g., a server database? Alas, no. But they will get many of them--SAS/ACCESS is increasingly sophisticated in which bits of queries it pushes to the db engine for execution.

At least as important, you have the freedom to invest in whatever back-end technology you like and can support. That preserves a domain of site freedom that I think is absolutely crucial to keeping sites engaged and happy. Staff at the sites are reallly the only ones who know what their loads are, and what resources are available to them. Are oracle DBAs hunted to extinction in your area (or just too damned expensive)? Go with Netezza. Licensing for Teradata too dear? Use MySQL. Does your parent org have a site license for DB2, and a ton of excess DB2 DBA capacity? Bob's your uncle.

Or maybe you're not feeling much pain with plain old SAS datasets--that too is an option of course.

So I say, let's communicate and share best practices of course (and we haven't done enough of that) and advise one another about what does/doesn't seem to work. But I don't see any gain to be had by invading the province of the SDMs and mandating some particular architecture. We've all got SAS, SAS is the interface. Apart from that, free to be you and me.

Just because your remote users use SAS to access your VDW data does not mean your local users have to.

As mentioned above, local users are the ones who suffer most from poor query performance (and they likely know where your desk is). If SAS' performance overhead is too costly to your local users, or they just prefer another tool, and you've taken the plunge into a server database, there is no reason not to offer these users access via that tool (apart from cost of course).

So--probably there's something I'm not thinking of, that makes some non-SAS language or environment more of a contender for VDW front-end. Tell me what it is in the comments.

Fundamentals

In which we ponder some very basic questions.

What are the goals of the VDW? Why does it exist?

The primary reason the VDW exists is to ease the burden of multi-site health-data-based research. By putting health data in a common format and specifying standard access methods, the VDW allows programming work done at one site to be used by the staff at most any other VDW-implementing site. The result is significant savings in cost and time spent on programming work.

In addition to that primary goal, the VDW has also had a couple of very nice side effects:

• At many sites, because the VDW offers a comparatively simple interface to its data to local programmers, they can be more productive even for local-only work.
• It has enabled program code to be reused across sites in addition to over time. This allows an economy of scale that should offer us better code--because it's been used and results scrutinized by many people on many different projects.

That first side effect has a nice side effect of its own: the additional scrutiny that local-only use gives results in additional vetting and defect detection. Sites whose local users use their VDW files almost certainly have better-quality files because bugs are discovered (and we presume, fixed) more quickly.

What sorts of things can be "in the VDW"?

In general, the VDW as a system consists of datasets and program code that manipulates those datasets. Datasets fall into 2 categories: those that hold substantive data (e.g. enrollment; utilization), and lookup or reference datasets (e.g., the datasets of chemotherapy-signifying codes that the CRN has put together.)

Code is generally SAS programs--either macros that get incorporated into the standard macro library, or as freestanding .sas programs that do a specific task.

What does it mean for a dataset to be "in the vdw"? What should it mean?

First and foremost, putting a dataset in the VDW means that we expect the dataset to support ongoing programs of research--not just individual projects. The VDW is infrastructure--at a level over projects.

While we hope and expect that nearly all projects can benefit from the VDW to some extent, and that some portion can even be accomplished exclusively with VDW data, we do not expect that every project will be accomplishable using just VDW data. We do not consider striving to be the end-all-be-all of research data to be an attainable goal. Implementing sites should not discard their indigenous data (or programmers ;-).

More practicaly, having a dataset in the VDW means that some group of people (technical and scientific) want the dataset to exist enough to form a workgroup, and:

• articulate the sorts of uses they imagine for the data,
• hammer out a spec (and optimally full-on implementation guidelines)
• answer questions from implementers (e.g., construe the spec)
• write qa program(s) and verify implementations, and
• Sites are able and willing to implement, update, document and support the files.

If we knew that a given dataset (or variable) was not getting used, would it make sense to drop it from the VDW?

That is certainly worth thinking about. I'm looking at you, Census.

What makes a good vdw spec?

A good VDW dataset specification is implementable. The best indication of this is that it has in fact been implemented--preferably at more than one site. You never really know what issues and questions you will run into until you actually go to implement. Many directions that seem clear and complete from your armchair raise knotty questions once you dig in to your indigenous data and see exactly what you have.

A good VDW spec is specific enough to give clear guidance to implementers as to how to wrangle their indigenous data into the spec, and to users as to what they should expect to find.

On the other hand, a good VDW spec is not unnecessarily specific--does not add requirements that do not serve the intended use of the data. There are many details of a dataset that do not really bear on its use. Variable labels and formats are easily in this category. I would argue that in most cases variable lengths are similarly of no import to end-users. In some cases (specifically, MRN, which is designed never to leave an implementing site) even variable type may not matter to users.

A good VDW spec preserves detail available in indigenous data without making it impossible for sites with less available detail to implement. So for examplethe old-timers among us will recall the time when the enrollment spec called for one record per member per month, and bore only the year and month. If a record existed, it meant that Joe Blow was enrolled for some portion of that month, and that was it.

Compare the new spec, which allows for pinpoint dates of enrollment & disenrollment. Now, sites that have that level of detail can include it, and those who only have month-level granularity can document that, and use first/last day of month in the enr_start and enr_end variables.

This can be a neat trick of course--there's a real tension between accomodating full detail and still giving a uniform interface across sites w/different levels of detail.

A good VDW spec uses relevant industry standards as much as possible--for example CPT, LOINC, NDCs, etc. Coming up with our own codesets for these sorts of things is a burden that we should try to avoid wherever possible.

A good VDW spec follows a coherent set of conventions for things like variable names (CamelCase or under_scores?) and missing value handling. This is an area where we could use some work, frankly--we have not paid enough attention to it in the past.
What are your thoughts on these questions? Please comment!