Tag Archives: Shapefile

JSON to Shapefile

31 Jan

Toronto has a bunch of open data – which is awesome! Someone I follow on Twitter was looking to convert some parking JSON to a Shapefile. I haven’t used it much, but python has a JSON module in 2.7 and SimpleJSON in 2.5. And of course, writing shapefiles is simple with shapefile.py.

JSON is a dictionary. Once it is loaded in a data variable, you can call items by the key. So for the carparks, I could call: data[“carparks”][i][“id”] or data[“carparks”][i][“lat”], where i is the index or the array. Then iterate, while i < len(data[“carparks”]).

Here is the full code:

import shapefile
import json

json_data=open(‘greenPParking.json’)
data = json.load(json_data)

w=shapefile.Writer(shapefile.POINT)
w.field(“id”)
w.field(“address”)
w.field(“lat”)
w.field(“lng”)
w.field(“rate”)
w.field(“rate_half_hour”)
w.field(“carpark_type”)
w.field(“carpark_type_str”)
w.field(“capacity”)
w.field(“max_height”)
w.field(“payment_options”)

i=0 # should be changed to — while (i < len(data[“carparks”]))
while (i<243):

w.point(float(data[“carparks”][i][“lng”]),float(data[“carparks”][i][“lat”]))
w.record(data[“carparks”][i][“id”],data[“carparks”][i][“address”],data[“carparks”][i][“lat”],data[“carparks”][i][“lng”],data[“carparks”][i][“rate”],data[“carparks”][i][“rate_half_hour”],data[“carparks”][i][“carpark_type”],data[“carparks”][i][“carpark_type_str”],data[“carparks”][i][“max_height”],data[“carparks”][i][“capacity”],data[“carparks”][i][“payment_options”],data[“carparks”][i][“rate_details”])
i+=1

prj = open(“csvSHP.prj”, “w”)
epsg = ‘GEOGCS[“WGS 84”,’
epsg += ‘DATUM[“WGS_1984”,’
epsg += ‘SPHEROID[“WGS 84”,6378137,298.257223563]]’
epsg += ‘,PRIMEM[“Greenwich”,0],’
epsg += ‘UNIT[“degree”,0.0174532925199433]]’
prj.write(epsg)
prj.close()

w.save(“greenPParkingSHP”)
json_data.close()

Creating a Shapefile From a MongoDB Using Python

14 Nov

I am always looking for ways to store data and create shapefiles, or other filetypes I need, from it. Just picked up a book on MongoDB last night and have thrown together a quick example of how to enter some point data using Long and Lat and then retrieving the data and sending it to Shapefile.py. This is the first example, if I get around to it, the next example will query for 10 points near another then spit that out to a shapefile. But I need to walk before I run, so here are my first steps.

  1. Install Mongo
  2. Install pymongo
  3. Install Shapefile.py Check out this blog for more info – this guy is awesome, it is where I got the code to write the projection: GeoSpatial Python.
  4. Run mongod
  5. You can use mongo, a full JavaScript Shell, to enter data, but I used Python.
  6. Enter some data. I entered two points (35,-106) and (35.8,-106.8)
  7. write a python program to convert this data to a shapefile.

Here is the python program to retrieve these two points, write a shapefile and a PRJ:

from pymongo import Connection,GEO2D
import shapefile

db=Connection().geo
w=shapefile.Writer(shapefile.POINT)
w.field(“comment”)

for x in db.places.find():
w.point(float(x[“loc”][0]),float(x[“loc”][1]))
w.record(“Booya!”)

prj = open(“Web.prj”, “w”)
epsg = ‘GEOGCS[“WGS 84”,’
epsg += ‘DATUM[“WGS_1984”,’
epsg += ‘SPHEROID[“WGS 84”,6378137,298.257223563]]’
epsg += ‘,PRIMEM[“Greenwich”,0],’
epsg += ‘UNIT[“degree”,0.0174532925199433]]’
prj.write(epsg)
prj.close()

w.save(“mongoSHP”)

;

Now you should have a shapefile with 2 points in WGS84

Python, Point Shapefile and Qgis

23 Oct

I have been getting data in a CSV and bringing it to Qgis using ‘Import Delimited Text.’  I could probably write a Python script in Qgis to do this automatically, but I only recently started reading the Qgis Python Cookbook. I have some experience with a Python Library for shapefiles already, so I will show how to create a point shapefile using this library.

Download Shapefile.py. All my maps are in WGS84, making this very easy. Open Notepad or your IDE and write the following:

import shapefile
w = shapefile.Writer(shapefile.POINT)
w.point(-106.505325, 35.069018)
w.point(-106.509378, 35.067741)
w.point(-106.527414, 35.108816)
w.field(‘type’)
w.field(‘street’)
w.record(‘res burg’,’central’)
w.record(‘auto burg’,’menaul’)
w.record(‘auto theft’,’eubank’)
w.save(‘point’)

I have three points in Albuquerque, with two fields: type and street. Running this file will create a shapfile without a PRJ. That’s alright, open the shapefile in Qgis. You should be asked for the projection. Choose WGS 84. It will display correctly. Now the points are on the map.

Right click on the shapefile in the table of contents and save as… Make sure the CRS is set to WGS 84.

Now when you open the shapefile you will not need to specify a projection.

Simple, right? Well, I hate hard coding the coordinates. I want to read from a file and everyone seems to send me data in Excel files so I downloaded XLRD and XLWT to read and write Excel files.

Here is how to add two points from an Excel Spreadsheet – you will want to run this as a loop to load all the data instead of hard coding it again. I will leave that up to you. (UPDATE: I can’t leave you hanging. Here is a new post with the code for looping). Here is the code to read the Excel file and write the shapefile.

book = xlrd.open_workbook(“test.xls”)
sh = book.sheet_by_index(0)
w = shapefile.Writer(shapefile.POINT)
w.point(sh.cell_value(rowx=0, colx=0), sh.cell_value(rowx=0, colx=1))
w.point(sh.cell_value(rowx=1, colx=0), sh.cell_value(rowx=1, colx=1))
w.field(‘type’)
w.field(‘street”)
w.record(sh.cell_value(rowx=0, colx=2),sh.cell_value(rowx=0, colx=3))
w.record(sh.cell_value(rowx=1, colx=2),sh.cell_value(rowx=1, colx=3))
w.save(‘fromExcel’)

Is this any easier than importing a CSV and creating the shapefile? No. But what it lets me do is schedule the Python file to run at intervals and update the shapefile even if I am at home sleeping. When graveyard shift comes in, they can see the latest data without me having to create it. It also allows me to add other functionality and python scripts.

I can filter the CSV by type or by location for example. Better yet, when a point in the shapefile exceeds some value, I can send a text to my phone – like this:

import shapefile
import xlrd
import smtplib
book = xlrd.open_workbook(“realtime.xls”)
sh = book.sheet_by_index(0)
w.record(sh.cell_value(rowx=1, colx=2))
w.save(‘writeANDreadANDshapefile’)

x = sh.cell_value(rowx=1, colx=2)
if x > 100:
message = “At Point: “+ str(sh.cell_value(rowx=1, colx=0)) +”,” + str(sh.cell_value(rowx=1, colx=1)) +”Levels equal: “+str(sh.cell_value(rowx=1, colx=2))
server = smtplib.SMTP( “smtp.gmail.com”, 587 )
server.starttls()
server.login( ‘YourAccount@gmail.com’, ‘Your Password’ )
server.sendmail( ‘From Phone Number@mms.att.net’, ‘ToPhoneNumber@mms.att.net’, message )

else:
print ‘Levels are normal’

All the concatenation of Message =”….” will create a string that says: “At Point (-106.4567, 35.3445) Levels Equal: 125” but will fill in the numbers using the actual coordinates and value.

There are programs – like CrimeView – that do this sort of thing automatically. But the way I showed was free and can be modified as you wish. Trying modifying the code of a proprietary software solution….Good Luck and Happy Hacking.