Back in the late 2000's, I took the terrifying step of creating folders on my computer to start pursing my formal dissertation research. Around the same time, I realized that my system for organizing my paper files had become a sandbag. The physical compartments I was using to segregate "different" aspects of my work were hurting my ability to see and explore the overlapping areas of several inter-connected problems. I tore everything apart and put it back together again so the overall structure was different, the grains of information were different, and the "bins" were collapsed into a single well that I could draw from. In order to stop blindly analyzing the different parts of the elephant and start trying to understand the whole animal, you first have to understand  that you're looking at pieces of a much larger puzzle.

It was more of a strategy than an epiphany. 

Last week I got into the nitty-gritty of a SEAC paper I'm writing with David G. Anderson (University of Tennessee). We're using various large datasets to try to describe and interpret patterns of change in archaeological remains that could be related to changes in the size, structure, and distribution of human populations in the Eastern Woodland during the Late Pleistocene and Early Holocene.  

As I started pulling together information (from PIDBA, DINAA, and my ongoing radiocarbon compilation) and thinking about how to organize it, I realized that keeping the databases separate was both a logistical hassle and an analytical problem. I invested in dumping all the information into a single relational database that we can use for this paper and that I'll continue to update in the future. I've been calling it "Megabase" in my head. So that's what it is until it gets a better name.

​Here is an illustration that I'll briefly discuss:

On the left is what I'm building now. I used GIS to generate a listing of "center" UTM coordinates (n=2097) for every county in the eastern US (everything east of the first tier of states west of the Mississippi River) and much of eastern Canada. I'm calling that the "County Core." That coordinate list lets me easily create a spatially-reference file for whatever other information I want from any of the other databases without needing to know the exact locations of archaeological sites.  Making a county-level map of all eastern radiocarbon dates in the database (9,533 and counting) in the eastern US is just a matter of a few button clicks in Access, Excel, and GIS. The same is true of the PIDBA data, the Kirk Project data, the household archaeology data, and the DINAA data. 

The Megabase of Today will be fine for the SEAC paper and for the near future. It will be able to do a lot. Ideally, however, the Megabase of the Future will have DINAA serving as both a "router" for data that is attached to a Smithsonian Trinomial and an analytical tool in its own right. One issue is that not all states are currently participating (and therefore not all Smithsonian Trinomials -- the "addresses" for sites -- are in the system).  Another issue is that the site forms (and therefore the site information that is collected and stored) differ by state. To reach its full potential, DINAA data will have to be supplemented by additional data about the materials recovered from sites, how sites were recorded, etc. Ensuring that we're making "apples to apples" comparisons will be a significant chore -- DINAA currently has information on somewhere in the neighborhood of half a million sites. You can't just sit on your couch and cross-check all that.

I know enough to be dangerous with a computer, but I'm not sufficiently sophisticated to know the nuts-and-bolts options for building the Megabase of the Future. In 2015 we did a sort of "proof" of concept to demonstrate that the EWHADP and DINAA could be linked together. I'm not sure if that is they way to go or not. Perhaps there's something that can be done with blockchain technology -- it sure sounds cool.

Anyway, I'm going to get the Megabase of Today functional in time to do the analysis for the SEAC paper we'll give in a month. If you're interested in talking about the Megabase of the Future, please let me know.

I just returned from several days at the DINAA (Digital Index of North American Archaeology) workshop that is happening at Indiana University South Bend this week.  The first two years of the DINAA project have focused on building a comprehensive, accessible database of archaeological sites in Eastern North America.  What the PIs and their core team (Josh Wells, David Anderson, Eric Kansa, Sarah Kansa, Steve Yerka, R. Carl DeMuth, Kelsey Noack Myers, and Thad Bissett) have accomplished to date is pretty remarkable:  in the pilot phase of the project, the team has assembled and made available primary data on over 340,000 recorded archaeological sites from ten states (with more data on the way).  This endeavor required navigating numerous technical, logistical, and political challenges.  What a great job they've done.  Bravo!

The DINAA project will benefit the archaeological community (and other constituencies) in a number of ways.  Some of those are obvious now, and some will become apparent as we become able to think in practical terms about the power and potential of a large, unified dataset that integrates and crosscuts the traditional (i.e., state-level) territories within which archaeological site data are managed. 

The benefit that I am most excited about is the potential of DINAA to act as a "bridge" among otherwise disconnected datasets. The key to this inter-linking is DINAA's granularity:  by making the archaeological site number the primary means by which information is organized, any datasets that reference a site number could be inter-linked through DINAA regardless of what primary information about the site is held by DINAA.  I'm compiling data on prehistoric house structures, for example, that could be linked, through DINAA, to other datasets using the key attribute of "site number."  Imagine being able to click on the site number associated with a structure in the Eastern Woodlands Household Archaeology Data Project (EWHADP) database and being led to a record in DINAA that provides links to a database of radiocarbon dates, or a spreadsheet of feature contents, or images from museum collections or field note archives, or bibliographic references for reports, dissertations, or academic papers that are also associated with that site number.  Or imagine being able to make a query for floral remains from Middle Woodland features within a 100 km radius of the site with that structure.  To say that that kind of inter-linking would be a powerful tool for research and scholarship is a great understatement.  As I argued in a presentation via Skype to the DINAA workshop that was held at the University of Tennessee in March, inter-linking of datasets would: (1) allow us to greatly expand the scale of questions we can address; (2) allow us to gather data for addressing those "big scale" questions much more efficiently; and (3) be a catalyst for developing and testing new interpretations of the past.  Engineering a system of links to connect diverse datasets would be a game changer.

At the workshop this week, we spent some time thinking about ways to actually accomplish such a linking.  The structure data I've been gathering is a good "test case" primarily because it spans the same area as the DINAA data and includes information from many sites (and is also open and freely accessible).  Information needs to flow both ways.  First, when a site record is called up in DINAA, it should be able to make a call to the EWHADP (and any other linked datasets) to see if there are records associated with that site number.  Second, site/structure records in the EWHADP should include a pointer to the appropriate record in DINAA.  The second part is simple: since the URLs associated with site records in DINAA are "stable," I can just put hyperlinks in my database that point to the DINAA record.  The second part is a little trickier.  Our first step was to put the current (as of March 2014) EWHADP database on GitHub (here) so that it would be open and accessible.  GitHub automatically tracks when changes are made to a file.  The next step (I think) will be to configure the GitHub page so that it sends a message to DINAA when the dataset changes.  This will allow the records in DINAA to be updated as new records are added to the EWHADP. 

It is the linking mechanisms that are important.  The EWHADP data do not become a "part" of DINAA but are simply referenced by DINAA.  I maintain control and responsibility for the EWHADP part of the equation.  This is important because compilation of that dataset is ongoing:  it's a database of records that I'm still collecting rather than something like a static set of measurements that relate to a single assemblage or site.  I hope others who are developing similar datasets think about how we might link them all together through DINAA.  The corpus of scholarship relevant to archaeological work in this part of the world is simply too large and diverse to live in a single place.  A distributed approach using a "bridge" such as DINAA to link datasets is going to be much more effective and useful than trying to house everything in one central place.

We all have a lot to gain by supporting the construction of this tool.  It is going to unleash the potential energy stored in the work we've already done and provide a "living" structure that will significantly increase our capacities to find, share, utilize, and build on archaeological information.  The DINAA project needs to move forward in a big way.