Tag Archives: NoSQL

Graph Database and Albuquerque Bus Stops: Neo4j with py2neo

15 Apr

I have been slightly obsessed with the question: “How do you define network service areas client-side on a map.” I know it needs a networked data set and something to do with the Djikstra algorithm (Yes, we could just use an ESRI REST service but there is not one available yet – I will ask the City). After looking at JavaScript implementations of NetworkX, I stumbled upon graph databases, most notably Neo4J.  A networked data set is a graph. Guess what, it has Djikstra built-in, so I must be on the right path. I installed it and added a fake social graph using py2neo. That allowed me to make sure I could do a few things:

  • Add a node
  • Add a relationship
  • add attributes

Now it was time to start with some real data.

My first test was to load Albuquerque Bus Stops for a single route. Here is what I have in my database.

Bus Stops for Route 766. No Relations added yet.

Bus Stops for Route 766. No Relations added yet.

The image above was generated by calling the City of Albuquerque REST Endpoint for bus stops, parsing the response, and putting it in to Neo4J. The image is a view from the DB Manager. The code to do this is below.

from py2neo import Graph
from py2neo import Node, Relationship
from py2neo import authenticate
import urllib2
import json

authenticate(“localhost:7474″,”myUserName”,”myPassword”)
graph=Graph()
graph.delete_all()

url=”http://coagisweb.cabq.gov/arcgis/rest/services/public/fullviewer/mapserver/22/query?where=ROUTE=’766’&f=json&outFields=*&outSR=4326″
rawreply=urllib2.urlopen(url).read()

reply=json.loads(rawreply)

for x in reply[“features”]:
graph.create(Node(“stop”,route=x[“attributes”][“ROUTE”],direction=x[“attributes”][“DIRECTION”],street=x[“attributes”][“STREET”],intersection=x[“attributes”][“NEAR_INTER”],lat=x[“geometry”][“y”],long=x[“geometry”][“x”]))

Notice there are no Relationships! This is crucial if we will ever walk the network. I have manually added on, seen in the image below.

San Mateo links to Louisianna.

San Mateo links to Louisianna.

The code for this is:

rel=Relationship(graph.node(42),”Next”,graph.node(41))

graph.create(rel)

I need to think about how to automate the relationship creation based on stop order and direction (there are stops on both sides of the street). Then, I will need to figure out how to make a node have relationships to other routes. For example, many stops are connected to the 777 route and I do not want a separate node for each. I want one with a property showing routes.

Well, a start to say the least. It has been fun learning about graph databases and if GIS doesn’t interest you, you could map your social network and walk it.

MongoDB, pymongo and GridFS

11 Sep

It has been a while since I’ve done anything with MongoDB-I changed jobs and don’t get to code much anymore. I had the urge to learn more and was interested in storing files in MongoDB using GridFS. I googled, read StackOverflow and MongoDB in Action. The problem was most info was storing text as a file or even when storing a file, the code got the _id when executing the put. So of course get is easy, you have the id. Figuring how to get after the fact was where I got stuck, also had to switch file operations to binary. Here is what I have for putting a file in MongoDB and how to retrieve it later.

from pymongo import MongoClient
import gridfs

client=MongoClient()
db=client.mytest
data=open(“image.png”,”rb”)
fs=gridfs.GridFS(db)
thedata=data.read()
stored=fs.put(thedata,filename=”inmongoimage”)

To get it back in the same code you call:
out=fs.get(stored).read()

This works because stored has the _id of the put operation. But what if I need to retrieve in different code from the inserts? Here is how I got it out with some extra code for info:
imports….
client=MongoClient()
db=client.mytest
fs=gridfs.GridFS(db)

filelist=fs.list()
#returns the file names stored

fileone=filelist[0].encode(“ascii”,”ignore”)
#returns a string of the first file name

outdata=fs.get_version(fileone,”rb”).read()
output=open(“somefile.png”,”wb”)
output.write(outdata)
output.close()

Put allows for more metadata than just the filename:
fs.put(thedata,filename=”file.jpg”,field=”string of text”,anumberfield=52)

To find on a different field:
fs.get_version(anumberfield=52)

MongoDB in QGIS

28 Dec
Displaying my MongoDB Data

Displaying my MongoDB Data

I recently installed the MongoDB Plugin for QGIS. The plugin allows you to connect to a MongoDB and load your data. I stored point data – long and lat – for all the public art in Albuquerque. Using the plugin, I connected to my MongoDB and loaded them.

I had trouble getting the plugin to install. My fix was to install pymongo and BSON to my local Python2.7 then copy it from the site-libs to the QGIS directory (C:\Program Files\Quantum GIS Lisboa\apps\Python27\Lib\site-packages). then I had to turn on the plugin under manage plugins.

This did the trick. Then I fired up my mongod, loaded QGIS and connected using the defaults. It prompted me for a Database and then a Collection. Looks like it requires a GEO2D. You can see my data in the image above.

Can’t wait to start playing….