Category Archives: Uncategorized

FracFocus removes PDF detailing SQL Server BAK file restore and import to Microsoft Access database.

As brought to my attention by “w”, it appears that FracFocus.org has removed the hyperlink (first link in second paragraph of this page) to its PDF document detailing how to restore the SQL Server BAK file into an SQL Server database and once that’s done, its subsequent conversion via import into a Microsoft Access database.  This, unfortunately, means that the average analyst/citizen is left with little, if any, guidance as to how to get the data within the BAK file into any format suitable for analysis.

FrackingData.org will endeavor to provide an alternative document and/or batch script, as soon as possible,  that will fill in this gap until FracFocus.org restores the original PDF document or an updated version, if so appropriate.  This, in this author’s humble opinion, is not “a good thing” in that doing such, especially without notice or commentary on the affected page, is a breach of transparency, both on notification and on access to the data itself.

Khepry Quixote
27 June 2016

Status Update 2016-06-23: FrackingData_FracFocusRegistry 2016-06 Files Uploaded

As of 23 June 2016, various files (e.g. SQlite, CSV, and PgSQL) derived from FracFocus.org’s June 2016 FracFocusRegistry have been downloaded, extracted, transformed, loaded, archived, and uploaded to the frackingdata.info/downloads site and their respective links also posted to FrackingData’s FracFocus Data Page .

This time, FracFocus posted their SQL Server backup on 20 June 2016, almost a month later than its previous posting of 23 May 2016.

Once again, of significance this time was that the download of the files from the FracFocus.org website and their subsequent extract, transform, load, archiving, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, archive, and export process.  In addition, the batch script now uses WinSCP to automatically upload the files in question to the http://frackingdata.info/downloads page.

When this Windows batch file is sufficiently stable, and I’ve soft-coded the data-cleansing views into the script itself,  I’ll post a link to it in the Source Code section of this blog.  Soft-coding of the data-cleansing views is the last hurdle to publishing this script.

Khepry Quixote 2016-06-23

Earthquakes: Latest (1898-01 thru 2016-05) Reverse-Geocoded Files Posted to frackingdata.info/downloads

As I promised earlier, I’ve downloaded earthquakes from NCEDC’s web site (1898 to date), reverse-geocoded them via GeoNames and K-D Trees (thereby obtaining their country, state, county, and city/village values), archived the resulting files via 7-ZIP and uploaded both the CSV and SQLite datasets to:

I have authored a program in Python 3 that reverse-geocodes (via GeoNames and K-D Trees) the lat/longs into their respective countries, states, counties, and cities/villages.  I will post a link to the open-source project shortly once I’ve vetted its license and repository.  The program processes nearly 3 million rows in approximately 240 seconds.

Status Update 2016-03-14: FrackingData_FracFocusRegistry 2016-02 Files Uploaded

As of 14 March 2016, various files (e.g. SQlite, CSV, and PgSQL) derived from FracFocus.org’s February 2016 FracFocusRegistry download have been uploaded to frackingdata.info/downloads site and their respective links posted to FrackingData’s FracFocus Data Page .

The substantial delay between the last posting of the transformed FracFocusRegistry download in January and this one in March was mostly due to FracFocus NOT posting anything until I reminded them via Twitter as of the evening of 2 March 2016.  This tardiness is becoming a pattern.  In times past I made quiet entreaties for them to post their data, however as I’m perceiving a pattern of semi-regular tardiness I’ll be admonish them in public from hereon.

Once again, of significance this time was that the download from the FracFocus Data Download website and subsequent extract, transform, load, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, and export process.

When this Windows batch file is sufficiently stable, I’ll post a link to it in the Source Code section of this blog.

Khepry Quixote 2016-03-14

Initiatives Page Created

The Initiatives page, visible in the menu, will list and track the progress of upcoming initiatives concerning this site.  Presently, there are three main initiatives: 1) render GUID keys to INT keys thereby reducing the upload/download size of files; 2) finalize the BAT or Python 3.4 scripts to download, extract, and transform the FracFocusRegistry ZIP file automatically into CSV, SQLITE, and PostgreSQL files; 3) Automate the push of the INT key CSV files to open-data Google Fusion tables so that analysts can access the data via Tableau, R, or SAS.

Status Update 2015-12-18: FrackingData_FracFocusRegistry 2015-11 Files Uploaded

As of 18 December 2015, various files (e.g. SQlite, CSV, and PgSQL) derived from FracFocus.org’s November 2015 FracFocusRegistry download have been uploaded to frackingdata.info/downloads site and their respective links posted to FrackingData’s FracFocus Data Page .

Once again, of significance this time was that the download from the FracFocus Data Download website and subsequent extract, transform, load, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, and export process.

When this Windows batch file is sufficiently stable, I’ll post a link to it in the Source Code section of this blog.

Khepry Quixote 2015-12-18

Status Update 2015-11-24: FrackingData_FracFocusRegistry 2015-10 Files Uploaded

As of 24 November 2015, various files (e.g. SQlite, CSV, and PgSQL) derived from FracFocus.org’s October 2015 FracFocusRegistry download have been uploaded to frackingdata.info/downloads site and their respective links posted to the FrackingData’s FracFocus Data Page .

Of significance this time was that the download from the FracFocus Data Download website and subsequent extract, transform, load, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention.  This automated method shaved hours from the extract, transform, load, and export process.

When this Windows batch file is sufficiently stable, I’ll post a link to it in the Source Code section of this blog.

Khepry Quixote 2015-11-24