Tag Archives: JSON

Albuquerque Live Bus Data as JSON

27 Jan

In my last post on the City of Albuquerque live bus data I showed how to parse the KML straight from the server in JavaScript. In this example, I will show you how to create your own REST endpoint for Route 766.

Classic ASP

Using classic ASP, because why not, I made a request to the server and read in the KML file.

url = “http://data.cabq.gov/transit/realtime/route/route766.kml”
set xmlhttp = CreateObject(“MSXML2.ServerXMLHTTP”)
Set xml = Server.CreateObject(“Microsoft.XMLDOM”)
xmlhttp.open “GET”, url, false
xmlhttp.send “”
xmlDoc=xmlhttp.responseText

The xmlhttp.responseText will hand back the KML file (KML is just XML) as text formatted as XML.

<?xml version=”1.0″ encoding=”UTF-8″?>
<kml xmlns=”http://www.opengis.net/kml/2.2″&gt;
<Document>
<Name>Route 766</Name>
<Style id=”NoDir”>

To parse it by nodes, we need to convert it to XML (though you could parse it using split() or a combination of other methods.

xml.async = False
xml.loadXML(xmlDoc)

To make the service return JSON, we will need to loop through all the Placemark/Point nodes and pull out the coordinates. We grab the node then setup our counters.

Set nodes = xml.SelectNodes(“//Placemark/Point”)
count=0
total=nodes.length-1

Before entering the loop, we need to setup our JSON string

Response.ContentType = “application/json”
response.write(“[“)

Finally, we can loop through and print each coordinate value in {}. Then we close the JSON string and empty out our variables.

For Each node In nodes
response.write “{”
For Each child in node.ChildNodes
Response.Write “”””& child.nodeName & “””: “”” & child.Text & “”””
Next
if count=total Then
response.write “}”
else
response.write”},”
count=count+1
End if
Next
response.write(“]”)
Set nodes = Nothing
set xmlhttp = nothing

Here is what the service returns

[{“coordinates”: “-106.65785,35.08641”},{“coordinates”: “-106.58566,35.07783”},{“coordinates”: “-106.56848,35.07648”},{“coordinates”: “-106.64804,35.0838”},{“coordinates”: “-106.69363,35.08368”},{“coordinates”: “-106.70488,35.0813”},{“coordinates”: “-106.57089,35.10303”}]

The KML is still ugly – mostly because the <description> node has a table as a child node and getting each value out of the table is annoying. Please, City of Albuquerque, please change the KML to not have tables but nodes for <Vehicle Number>, <Speed., <Msg Time> and <Next Stop>.

Client Side Consumption of the REST Service

With the service up and running, we can query it using AJAX on the client side and draw a map. In the client side JavaScript, we can take httpresponseText and parse it to JSON. Iterating through the length, we grap the coordinates and split them on the “,”. Then add them to the map. We could improve our JSON service to split the coordinates for us and write out an X and Y attribute.

var url = “http://dmdview/paul/xml.asp&#8221;;
http=new XMLHttpRequest();
http.open(“GET”, url, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {//Call a function when the state changes.
if(http.readyState == 4 && http.status == 200) {
console.log(http.responseText);
var result= JSON.parse(http.responseText);
for(x=0;x<Object.keys(result).length;x++){
xy=result[x].coordinates;
coords=xy.split(“,”);
markers.addLayer(L.marker([coords[1],coords[0]]))
}//end for
markers.addTo(map);}}
http.send();

Now you have a JSON service that can be replicated for each bus route.

Advertisements

Database Queries Using GIS: The ESRI REST API Without Geography

6 Jan

When people think of GIS they think of maps. But GIS is about data and if we think about it as data it is not special. In this post, I will show you how to use your GIS data as a database and nothing more. I am going to build a website that displays a table of all the public art in Albuquerque with the artist, title and a link to an image of the piece.

First, go to the Albuquerque Public Data website and select Public Art. You will see a directory listing. Choose the last option: PublicArtREST. You will then see a bunch of the ArcServer information for the service. This URL could be used to load the data in to a map for display, but we are only interested in the tabular data at this time. On the bottom of the page you will see the supported operations. We are going to use the query operation. Click the query link and you will see a form. In the where box, type 1=1 and then click the button query (GET). a list of features will show up under the form. See the geographic data available as (x.y) coordinates? In the outfields box, type a *. Then set the return geometry option to False. Now press Query (GET) again. Now you have all the data from the database (there are x,y fields only because they are an actual column in the table. This is not the geometry from ESRI. You can tell because the projection changed – it is now in wkid:4326 and the ESRI version is in wkid:3857).

Now you know where the data lives and how to get it without the geographic portion. The next step is to grab it using AJAX. The key line in the AJAX query will be the parameters:

var params = “where=1=1&outfields=*&f=json”;

You used the first two parameters in the form. The 1=1 returns all the records(ArcServer limits the number of results so a large dataset may only return between 1000-2500 unless it is set for more). The outfield=* returns all the fields for each feature. The last parameter “f=JSON” is returning the JSON data and not the HTML underneath the form. This makes it easy to parse in JavaScript.

The complete code will create two DIV elements: one for the count of public art pieces; and one for the table of results. It will make an AJAX post request, count the records and write out the number, then create a table of the results by sticking each line in to an array then dumping it at the end. The result is below with the complete code after the image.

Table of Public Art with Link to Image

Table of Public Art with Link to Image

<html>
<head>
<title>PUBLIC ART</title>
</head>
<body>
<div id=”count”></div>
<div id=”data”></div>
<script>
var lines=[“<table border=’1′><tr><td>TITLE</td><td>ARTIST</td><td>IMAGE URL</td></tr>”];
var params = “where=1=1&outfields=*&f=json”;
var url = “http://coagisweb.cabq.gov/arcgis/rest/services/public/PublicArt/MapServer/0/query&#8221;;
http=new XMLHttpRequest();
http.open(“POST”, url, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {
if(http.readyState == 4 && http.status == 200) {
theGeom= JSON.parse(http.responseText);
var key, count = 0;
for(key in theGeom.features) {
if(theGeom.features.hasOwnProperty(key)) {
count++;
}}
document.getElementById(“count”).innerHTML=”<h3>There are “+count+” pieces of Public Art in Albuquerque</h3>”;

for(i=0;i<count;i++){

lines.push(“<tr><td>”+theGeom.features[i].attributes.TITLE+”</td><td>”+theGeom.features[i].attributes.ARTIST+”</td><td><a href='”+theGeom.features[i].attributes.IMAGE_URL+”‘>Image</a></td></tr>”);}

lines.push(“</table>”);

document.getElementById(“data”).innerHTML=lines.join(” “);

}}

http.send(params);
</script>
</body>
</html>

Now you know how to use a GIS service to retrieve data without a map. You can refine your query to only pieces by a certain artist, or create a table that lists artists and the number of pieces they have in the collection.

Happy Hacking.

ESRI REST API: GeoCoding OR OpenStreet Map

10 Jun

UPDATE: OpenStreet Map code at bottom.

 

Saw a Tweet saying that the ArcGIS REST API for Geocoding is moving to a new URL. I use Google when I just needed a single Lat,Long and ArcPy to Geocode CSV’s, but I decided to check out the REST API and throw together a Python script from my old Google one – which I got from Foundations of Python Network Programming.

Not much to the script – pass parameters to a URL and grab the JSON. You can go to the HELP for more info about the parameters – like setting outfields=*;

Here is the script (it grabs the first result):

import urllib, urllib2, simplejson
import csv

param = {‘Address’: ‘400 Roma SE’,’City’:’albuquerque’,’Region’:’nm’,’Postal’:’87102′,’outFields’:’location’,’f’:’pjson’}
url = ‘http://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/findAddressCandidates?&#8217; + urllib.urlencode(param)
rawreply = urllib2.urlopen(url).read()
reply = simplejson.loads(rawreply)

print “Long: ”
print reply[“candidates”][0][“location”][“x”]

print “Lat: ”
print reply[“candidates”][0][“location”][“y”]

 

OpenStreet Map is almost the same, just change the URL and the parameters.

import urllib, urllib2, simplejson
import csv

param = {‘q’: ‘400 roma, albuquerque’,’format’:’json’,’addressdetails’:’1′}
url = ‘http://nominatim.openstreetmap.org/search?&#8217; + urllib.urlencode(param)
rawreply = urllib2.urlopen(url).read()
reply = simplejson.loads(rawreply)

print reply[0][“lat”]
print reply[0][“lon”]

 

JSON to Shapefile

31 Jan

Toronto has a bunch of open data – which is awesome! Someone I follow on Twitter was looking to convert some parking JSON to a Shapefile. I haven’t used it much, but python has a JSON module in 2.7 and SimpleJSON in 2.5. And of course, writing shapefiles is simple with shapefile.py.

JSON is a dictionary. Once it is loaded in a data variable, you can call items by the key. So for the carparks, I could call: data[“carparks”][i][“id”] or data[“carparks”][i][“lat”], where i is the index or the array. Then iterate, while i < len(data[“carparks”]).

Here is the full code:

import shapefile
import json

json_data=open(‘greenPParking.json’)
data = json.load(json_data)

w=shapefile.Writer(shapefile.POINT)
w.field(“id”)
w.field(“address”)
w.field(“lat”)
w.field(“lng”)
w.field(“rate”)
w.field(“rate_half_hour”)
w.field(“carpark_type”)
w.field(“carpark_type_str”)
w.field(“capacity”)
w.field(“max_height”)
w.field(“payment_options”)

i=0 # should be changed to — while (i < len(data[“carparks”]))
while (i<243):

w.point(float(data[“carparks”][i][“lng”]),float(data[“carparks”][i][“lat”]))
w.record(data[“carparks”][i][“id”],data[“carparks”][i][“address”],data[“carparks”][i][“lat”],data[“carparks”][i][“lng”],data[“carparks”][i][“rate”],data[“carparks”][i][“rate_half_hour”],data[“carparks”][i][“carpark_type”],data[“carparks”][i][“carpark_type_str”],data[“carparks”][i][“max_height”],data[“carparks”][i][“capacity”],data[“carparks”][i][“payment_options”],data[“carparks”][i][“rate_details”])
i+=1

prj = open(“csvSHP.prj”, “w”)
epsg = ‘GEOGCS[“WGS 84”,’
epsg += ‘DATUM[“WGS_1984”,’
epsg += ‘SPHEROID[“WGS 84”,6378137,298.257223563]]’
epsg += ‘,PRIMEM[“Greenwich”,0],’
epsg += ‘UNIT[“degree”,0.0174532925199433]]’
prj.write(epsg)
prj.close()

w.save(“greenPParkingSHP”)
json_data.close()

Albuquerque Registered Historic Places as GeoJSON in Leaflet

13 Dec
Albuquerque Registered Historic Places.

Albuquerque Registered Historic Places.

I will say it one more time: I hate that Albuquerque publishes data in KMZ. I took the Registered Historic Places file and converted it to GeoJSON. Then I put up this simple Leaflet map using the GeoJSON file. My website is at abqhistoric.educationalfacilityplanning.com and I put the GeoJSON inside a JavaScript file where it was assigned to a variable ‘abq’. Here is the GeoJSON.

D3.js and MongoDB

28 Nov

I have not been shy in my love of MongoDB. The honeymoon is not over. Now I want to graph and visualize my data from MongoDB. I just started looking at D3 – I’m coming to the party a bit late – and it is perfect for this task. I have thrown together a super simple, absurd even, example using a DB I had already populated and some left over CherryPy code. Let me say that reusing code is a good idea – when I can find what I’m looking for. It allows me to throw something together quickly. These are not production samples, but just for me to see that I can get it to work. Enough already. Here is a bar chart using D3 and MongoDB.

d3

A Bar Chart, that should be in SVG, from MongoDB data in D3.js

I have not put this on OpenShift yet, but may.  This bar chart is a simple python script using CherryPy and committing my favorite sin – passing HTML as a variable in a return. One reason to put it on OpenShift is so I can template it in Jinja.  Here is the code:

import cherrypy
from pymongo import Connection

class mongocherry(object):

def index(self):
db=Connection().geo
output =[]
output.append(‘<HTML><HEAD><TITLE>D3 and MONGODB</TITLE><script type=”text/javascript” src=”http://d3js.org/d3.v2.min.js”></script><style type=”text/css”>div.bar {display: inline-block;width: 20px;height: 75px;margin-right: 2px;background-color: teal;}</style></HEAD><BODY><h1>D3 and MongoDB</h1>’)
output.append(‘<script type=”text/javascript”>var dataset=[‘)

for x in db.places.find():
output.append(str(x[“loc”][0])+’,’)
output.append(‘0];’+”\n”+’d3.select(“body”).selectAll(“div”).data(dataset).enter().append(“div”).attr(“class”, “bar”).style(“height”, function(d) {var barHeight = d * 5;return barHeight + “px”;});</script></body></html>’)

i=0
html=””
while i<len(output):
html+=str(output[i])
i+=1

return html

index.exposed = True

cherrypy.config.update({‘server.socket_host’: ‘127.0.0.1’,
‘server.socket_port’: 8000,
})

cherrypy.quickstart(mongocherry())

The Python code prints out HTML that looks like this:

<HTML><HEAD><TITLE>D3 and MONGODB</TITLE>
<script type="text/javascript" src="http://d3js.org/d3.v2.min.js"></script>
<style type="text/css">
div.bar {
display: inline-block;
width: 20px;
height: 75px;
margin-right: 2px;
background-color: teal;}
</style>
</HEAD><BODY>
<h1>D3 and MongoDB</h1>
<script type="text/javascript">
var dataset=[35,35.8,38,39,30,31,31,31,33,25,33,0];
d3.select("body")
.selectAll("div")
.data(dataset)
.enter()
.append("div")
.attr("class", "bar")
.style("height", function(d) {var barHeight = d * 5;return barHeight + "px";});
</script></body></html>

Not much going on here, just a simple D3.js bar chart – not even done in SVG. The MongoDB part is in the variable dataset[..]. While writing the HTML, the python code loops through my mongodb with for x in db.places.find(): and it grabs the latitude of the data I have with x[“loc”][0] prints it out in the JavaScript variable dataset[]. I add a 0 at the end because I get a trailing comma. Sloppy, but oh well.

I return all the HTML and you get the page displayed at the top of this post. The cool part of marrying D3.js and MongoDB is JSON. I have JSON in my DB and D3 takes JSON.