As I promised earlier, I’ve downloaded earthquakes from NCEDC’s web site (1898 to date), reverse-geocoded them via GeoNames and K-D Trees (thereby obtaining their country, state, county, and city/village values), archived the resulting files via 7-ZIP and uploaded both the CSV and SQLite datasets to:

• FrackingData Reverse-Geocoded Earthquakes from 1898-01-01 to date (updated on an approximately monthly basis).
  • NCEDC_earthquakes_18980101_thru_20170221_reverse_geocoded_y_header.csv.7z
    • comma-separated values, approximately 75 MB in size. compressed, 391 MB expanded
  • NCEDC_earthquakes_18980101_thru_20170221_reverse_geocoded.sqlite.7z
    • SQLite 3 database, approximately 162 MB in size, compressed, 1.1 GB expanded

I have authored a program in Python 3 that reverse-geocodes (via GeoNames and K-D Trees) the lat/longs into their respective countries, states, counties, and cities/villages.  This is the link to the open-source Python 3 reverse-geocoder project.  The program processes nearly 3 million rows in approximately 240 seconds.

Khepry Quixote
21 Feb 2017

As of 15 Feb 2017, various files (e.g. SQlite, CSV) derived from FracFocus.org’s February 2017 FracFocusRegistry have been downloaded, extracted, transformed, loaded, archived, and their respective links posted to FrackingData’s FracFocus Data Page .

This time, FracFocus posted their SQL Server backup on 06 Feb 2017, about 3weeks later than its previous posting of 17 Jan 2017.

Once again, of significance this time was that the download of the files from the FracFocus.org website and their subsequent extract, transform, load, archiving, and exporting to CSV, SQLite, and PostgreSQL files was performed by a Windows batch script without human intervention. This automated method shaved hours from the extract, transform, load, archive, and export process.

When this Windows batch file is sufficiently stable, and I’ve soft-coded the data-cleansing views into the script itself,  I’ll post a link to it in the Source Code section of this blog.  Soft-coding of the data-cleansing views is the last hurdle to publishing this script.

Khepry Quixote 2017-02-15