Tag Archives: Graph

Graph Database and Albuquerque Bus Stops: Neo4j with py2neo

15 Apr

I have been slightly obsessed with the question: “How do you define network service areas client-side on a map.” I know it needs a networked data set and something to do with the Djikstra algorithm (Yes, we could just use an ESRI REST service but there is not one available yet – I will ask the City). After looking at JavaScript implementations of NetworkX, I stumbled upon graph databases, most notably Neo4J.  A networked data set is a graph. Guess what, it has Djikstra built-in, so I must be on the right path. I installed it and added a fake social graph using py2neo. That allowed me to make sure I could do a few things:

  • Add a node
  • Add a relationship
  • add attributes

Now it was time to start with some real data.

My first test was to load Albuquerque Bus Stops for a single route. Here is what I have in my database.

Bus Stops for Route 766. No Relations added yet.

Bus Stops for Route 766. No Relations added yet.

The image above was generated by calling the City of Albuquerque REST Endpoint for bus stops, parsing the response, and putting it in to Neo4J. The image is a view from the DB Manager. The code to do this is below.

from py2neo import Graph
from py2neo import Node, Relationship
from py2neo import authenticate
import urllib2
import json

authenticate(“localhost:7474″,”myUserName”,”myPassword”)
graph=Graph()
graph.delete_all()

url=”http://coagisweb.cabq.gov/arcgis/rest/services/public/fullviewer/mapserver/22/query?where=ROUTE=’766’&f=json&outFields=*&outSR=4326″
rawreply=urllib2.urlopen(url).read()

reply=json.loads(rawreply)

for x in reply[“features”]:
graph.create(Node(“stop”,route=x[“attributes”][“ROUTE”],direction=x[“attributes”][“DIRECTION”],street=x[“attributes”][“STREET”],intersection=x[“attributes”][“NEAR_INTER”],lat=x[“geometry”][“y”],long=x[“geometry”][“x”]))

Notice there are no Relationships! This is crucial if we will ever walk the network. I have manually added on, seen in the image below.

San Mateo links to Louisianna.

San Mateo links to Louisianna.

The code for this is:

rel=Relationship(graph.node(42),”Next”,graph.node(41))

graph.create(rel)

I need to think about how to automate the relationship creation based on stop order and direction (there are stops on both sides of the street). Then, I will need to figure out how to make a node have relationships to other routes. For example, many stops are connected to the 777 route and I do not want a separate node for each. I want one with a property showing routes.

Well, a start to say the least. It has been fun learning about graph databases and if GIS doesn’t interest you, you could map your social network and walk it.

Pandas and Python…..Oh My.

21 Dec

Crunching data and rearranging data in Python is cool, but I really need to visualize it. Nothing fancy, just a bar or line chart. I recently saw a D3 implementation in Python – awesome, but for now I just want to stick to Matplotlib. I grabbed a few books on scientific python and data in python. They seem to love IPython – the web notebook is pretty cool. Another tool that is often mention is Pandas. This is what I want to use – it uses Matplotlib for plotting. The one feature that caught my attention right away was the DataFrame. Think of it as a Excel spreadsheet, then if anyone asks about it, tell them it’s like data.frame() in R.  When I learn something, I like to start bare bones, then build it up with extra options and variations. I have put together some very minimal examples of plotting DataFrames and a Series in Pandas. From here you should have a good grasp of how to do more.

Plotting a Series.

Plotting a Series.

Plotting a Series requires a Series and a type of chart. Here is my code:

from pandas import Series
import matplotlib.pyplot as plt

b=[2,4,6,8,10]
a=Series(b,index=[‘a’,’b’,’c’,’d’,’e’])
Series.plot(a, kind=’bar’) #change to ‘barh’ for horizontal. Can also declare kind=’line’
plt.show()

Plotting a DataFrame

Plotting a DataFrame

 

Plotting a DataFrame is what I need the most in my work. Earlier I compared a DataFrame to an Excel spreadsheet. Here is what a DataFrame looks like:

Simple DataFrame

Simple DataFrame

Looking at the DataFrame and the Chart,  notice that each row plots as a group labeled by the index and columns. The DataFrame is created by passing a Numpy Array.:

a=np.array([[3,6,8,9,6],[2,3,4,5,6],[4,5,6,7,8],[3,6,5,8,6],[5,8,8,6,5]])
df=DataFrame(a, columns=[‘a’,’b’,’c’,’d’,’e’], index=[2,4,6,8,10])

To plot the chart, just call plot and pass a type. Here is the complete code:

from pandas import DataFrame
import matplotlib.pyplot as plt
import numpy as np

a=np.array([[3,6,8,9,6],[2,3,4,5,6],[4,5,6,7,8],[3,6,5,8,6],[5,8,8,6,5]])
df=DataFrame(a, columns=[‘a’,’b’,’c’,’d’,’e’], index=[2,4,6,8,10])

df.plot(kind=’bar’)
plt.show()

This is how I learned to use Pandas DataFrame and to plot my data. Knowing this, I felt much more comfortable looking at more advanced examples online.

D3.js and MongoDB

28 Nov

I have not been shy in my love of MongoDB. The honeymoon is not over. Now I want to graph and visualize my data from MongoDB. I just started looking at D3 – I’m coming to the party a bit late – and it is perfect for this task. I have thrown together a super simple, absurd even, example using a DB I had already populated and some left over CherryPy code. Let me say that reusing code is a good idea – when I can find what I’m looking for. It allows me to throw something together quickly. These are not production samples, but just for me to see that I can get it to work. Enough already. Here is a bar chart using D3 and MongoDB.

d3

A Bar Chart, that should be in SVG, from MongoDB data in D3.js

I have not put this on OpenShift yet, but may.  This bar chart is a simple python script using CherryPy and committing my favorite sin – passing HTML as a variable in a return. One reason to put it on OpenShift is so I can template it in Jinja.  Here is the code:

import cherrypy
from pymongo import Connection

class mongocherry(object):

def index(self):
db=Connection().geo
output =[]
output.append(‘<HTML><HEAD><TITLE>D3 and MONGODB</TITLE><script type=”text/javascript” src=”http://d3js.org/d3.v2.min.js”></script><style type=”text/css”>div.bar {display: inline-block;width: 20px;height: 75px;margin-right: 2px;background-color: teal;}</style></HEAD><BODY><h1>D3 and MongoDB</h1>’)
output.append(‘<script type=”text/javascript”>var dataset=[‘)

for x in db.places.find():
output.append(str(x[“loc”][0])+’,’)
output.append(‘0];’+”\n”+’d3.select(“body”).selectAll(“div”).data(dataset).enter().append(“div”).attr(“class”, “bar”).style(“height”, function(d) {var barHeight = d * 5;return barHeight + “px”;});</script></body></html>’)

i=0
html=””
while i<len(output):
html+=str(output[i])
i+=1

return html

index.exposed = True

cherrypy.config.update({‘server.socket_host’: ‘127.0.0.1’,
‘server.socket_port’: 8000,
})

cherrypy.quickstart(mongocherry())

The Python code prints out HTML that looks like this:

<HTML><HEAD><TITLE>D3 and MONGODB</TITLE>
<script type="text/javascript" src="http://d3js.org/d3.v2.min.js"></script>
<style type="text/css">
div.bar {
display: inline-block;
width: 20px;
height: 75px;
margin-right: 2px;
background-color: teal;}
</style>
</HEAD><BODY>
<h1>D3 and MongoDB</h1>
<script type="text/javascript">
var dataset=[35,35.8,38,39,30,31,31,31,33,25,33,0];
d3.select("body")
.selectAll("div")
.data(dataset)
.enter()
.append("div")
.attr("class", "bar")
.style("height", function(d) {var barHeight = d * 5;return barHeight + "px";});
</script></body></html>

Not much going on here, just a simple D3.js bar chart – not even done in SVG. The MongoDB part is in the variable dataset[..]. While writing the HTML, the python code loops through my mongodb with for x in db.places.find(): and it grabs the latitude of the data I have with x[“loc”][0] prints it out in the JavaScript variable dataset[]. I add a 0 at the end because I get a trailing comma. Sloppy, but oh well.

I return all the HTML and you get the page displayed at the top of this post. The cool part of marrying D3.js and MongoDB is JSON. I have JSON in my DB and D3 takes JSON.