Category Archives: Uncategorized

Status Update 2016-10-18: FrackingData_FracFocusRegistry 2016-10 Files Uploaded

As of 18 Oct 2016, various files (e.g. SQlite, CSV) derived from FracFocus.org’s October 2016 FracFocusRegistry have been downloaded, extracted, transformed, loaded, archived, and uploaded to the frackingdata.info/downloads site and their respective links also posted to FrackingData’s FracFocus Data Page .

This time, FracFocus posted their SQL Server backup on 17 Oct 2016, about 4 weeks later than its previous posting of 21 Sep 2016.

Once again, of significance this time was that the download of the files from the FracFocus.org website and their subsequent extract, transform, load, archiving, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, archive, and export process.  In addition, the batch script now uses WinSCP to automatically upload the files in question to the http://frackingdata.info/downloads page.

When this Windows batch file is sufficiently stable, and I’ve soft-coded the data-cleansing views into the script itself,  I’ll post a link to it in the Source Code section of this blog.  Soft-coding of the data-cleansing views is the last hurdle to publishing this script.

Khepry Quixote 2016-10-18

Please support the FrackingData.org IndieGoGo Campaign, now underway!

Please support the FrackingData.org IndieGoGo Campaign!  It’s time for FrackingData.org to move from an “after-hours” level of effort to the next level, as well as establish itself as a non-profit with a life beyond its founder, as the fight to obtain and post open-data and open-source relevant to fracking data disclosures is one that will likely last for many years, if not decades, to come.

Status Update 2016-09-21: FrackingData_FracFocusRegistry 2016-09 Files Uploaded

As of 21 Sep 2016, various files (e.g. SQlite, CSV) derived from FracFocus.org’s September 2016 FracFocusRegistry have been downloaded, extracted, transformed, loaded, archived, and uploaded to the frackingdata.info/downloads site and their respective links also posted to FrackingData’s FracFocus Data Page .

This time, FracFocus posted their SQL Server backup on 21 Sep 2016, about 5 weeks later than its previous posting of 15 Aug 2016.

Once again, of significance this time was that the download of the files from the FracFocus.org website and their subsequent extract, transform, load, archiving, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, archive, and export process.  In addition, the batch script now uses WinSCP to automatically upload the files in question to the http://frackingdata.info/downloads page.

When this Windows batch file is sufficiently stable, and I’ve soft-coded the data-cleansing views into the script itself,  I’ll post a link to it in the Source Code section of this blog.  Soft-coding of the data-cleansing views is the last hurdle to publishing this script.

Khepry Quixote 2016-09-21

Earthquakes: Latest (1898-01 thru 2016-09-11) Reverse-Geocoded Files Posted to frackingdata.info/downloads

As I promised earlier, I’ve downloaded earthquakes from NCEDC’s web site (1898 to date), reverse-geocoded them via GeoNames and K-D Trees (thereby obtaining their country, state, county, and city/village values), archived the resulting files via 7-ZIP and uploaded both the CSV and SQLite datasets to:

I have authored a program in Python 3 that reverse-geocodes (via GeoNames and K-D Trees) the lat/longs into their respective countries, states, counties, and cities/villages.  This is the link to the open-source Python 3 reverse-geocoder project.  The program processes nearly 3 million rows in approximately 240 seconds.

Khepry Quixote
11 Sep 2016

Earthquakes: Latest (1898-01 thru 2016-08) Reverse-Geocoded Files Posted to frackingdata.info/downloads

As I promised earlier, I’ve downloaded earthquakes from NCEDC’s web site (1898 to date), reverse-geocoded them via GeoNames and K-D Trees (thereby obtaining their country, state, county, and city/village values), archived the resulting files via 7-ZIP and uploaded both the CSV and SQLite datasets to:

I have authored a program in Python 3 that reverse-geocodes (via GeoNames and K-D Trees) the lat/longs into their respective countries, states, counties, and cities/villages.  I will post a link to the open-source project shortly once I’ve vetted its license and repository.  The program processes nearly 3 million rows in approximately 240 seconds.

Khepry Quixote
19 Aug 2016

 

Status Update 2016-08-19: FrackingData_FracFocusRegistry 2016-08 Files Uploaded

As of 18 Aug 2016, various files (e.g. SQlite, CSV) derived from FracFocus.org’s August 2016 FracFocusRegistry have been downloaded, extracted, transformed, loaded, archived, and uploaded to the frackingdata.info/downloads site and their respective links also posted to FrackingData’s FracFocus Data Page .

This time, FracFocus posted their SQL Server backup on 15 Aug 2016, about 4 weeks later than its previous posting of 14 July 2016 (an improvement, yeah!).  But, the ZIP archive was flawed in that it contained the July 2016 SQL Server backup instead of the Aug 2016 backup.  Sometime in the next two days, after I tweeted about the mistake, FracFocus updated the ZIP archive with the proper file.

Once again, of significance this time was that the download of the files from the FracFocus.org website and their subsequent extract, transform, load, archiving, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, archive, and export process.  In addition, the batch script now uses WinSCP to automatically upload the files in question to the http://frackingdata.info/downloads page.

When this Windows batch file is sufficiently stable, and I’ve soft-coded the data-cleansing views into the script itself,  I’ll post a link to it in the Source Code section of this blog.  Soft-coding of the data-cleansing views is the last hurdle to publishing this script.

Khepry Quixote 2016-08-19

FracFocus.org data download for Aug 2016 appears to be correct now.

Initially, the 15 Aug 2016 Microsoft SQL Server 2012 backup data download ZIP archive posted to their download page contained the SQL Server backup file for 14 July 2016.  Now, it appears that the backup file posted to their data download site is the actual one for August, 2016.

I’ve pulled the new backup file, processed it, and pushed up the CSV and SQLite files for 2016-08, to wit:

Use 7-Zip to unzip them and SQLiteStudio to examine the SQLite database.

Happy analyzing!

Khepry Quixote
18 Aug 2016