Tag Archives: Python

Graph Database and Albuquerque Bus Stops: Neo4j with py2neo

15 Apr

I have been slightly obsessed with the question: “How do you define network service areas client-side on a map.” I know it needs a networked data set and something to do with the Djikstra algorithm (Yes, we could just use an ESRI REST service but there is not one available yet – I will ask the City). After looking at JavaScript implementations of NetworkX, I stumbled upon graph databases, most notably Neo4J.  A networked data set is a graph. Guess what, it has Djikstra built-in, so I must be on the right path. I installed it and added a fake social graph using py2neo. That allowed me to make sure I could do a few things:

  • Add a node
  • Add a relationship
  • add attributes

Now it was time to start with some real data.

My first test was to load Albuquerque Bus Stops for a single route. Here is what I have in my database.

Bus Stops for Route 766. No Relations added yet.

Bus Stops for Route 766. No Relations added yet.

The image above was generated by calling the City of Albuquerque REST Endpoint for bus stops, parsing the response, and putting it in to Neo4J. The image is a view from the DB Manager. The code to do this is below.

from py2neo import Graph
from py2neo import Node, Relationship
from py2neo import authenticate
import urllib2
import json

authenticate(“localhost:7474″,”myUserName”,”myPassword”)
graph=Graph()
graph.delete_all()

url=”http://coagisweb.cabq.gov/arcgis/rest/services/public/fullviewer/mapserver/22/query?where=ROUTE=’766’&f=json&outFields=*&outSR=4326″
rawreply=urllib2.urlopen(url).read()

reply=json.loads(rawreply)

for x in reply[“features”]:
graph.create(Node(“stop”,route=x[“attributes”][“ROUTE”],direction=x[“attributes”][“DIRECTION”],street=x[“attributes”][“STREET”],intersection=x[“attributes”][“NEAR_INTER”],lat=x[“geometry”][“y”],long=x[“geometry”][“x”]))

Notice there are no Relationships! This is crucial if we will ever walk the network. I have manually added on, seen in the image below.

San Mateo links to Louisianna.

San Mateo links to Louisianna.

The code for this is:

rel=Relationship(graph.node(42),”Next”,graph.node(41))

graph.create(rel)

I need to think about how to automate the relationship creation based on stop order and direction (there are stops on both sides of the street). Then, I will need to figure out how to make a node have relationships to other routes. For example, many stops are connected to the 777 route and I do not want a separate node for each. I want one with a property showing routes.

Well, a start to say the least. It has been fun learning about graph databases and if GIS doesn’t interest you, you could map your social network and walk it.

Advertisements

Python Wrapper for Leaflet

20 Mar

I recently stumbled upon Folium – a python wrapper for leaflet. I was excited and it seemed to work well. I slowly ran in to problems and the pages loaded slow. I probably did something wrong on my end, but decided to write a simple wrapper on my own.

My wrapper is a python function for different Leaflet features such as map and marker. When you call each function, it writes a string to a file to generate the HTML. Below is my python code (pyLeaflet.py).

class l(object):

def __init__(self,path):
self.path=path
self.f=open(self.path,”w+”)
self.f.write(‘<html><head><title>Map From Python</title><link rel=”stylesheet” href=”http://cdn.leafletjs.com/leaflet-0.7.2/leaflet.css&#8221; /></head><body><script src=”http://cdn.leafletjs.com/leaflet-0.7.2/leaflet.js”></script><div style=”height:900px; width:900px” id=”map”></div><script>\n’)

def map(self, lat,long,zoom):
self.lat=lat
self.long=long
self.zoom=zoom
self.f.write(“var map = L.map(‘map’, {center: [“+str(self.lat)+”,”+str(self.long)+”], zoom:”+str(self.zoom)+”});\n”)
self.f.write(“L.tileLayer(‘http://{s}.tile.osm.org/{z}/{x}/{y}.png’).addTo(map);\n”)
def marker(self,lat,long, popup=””):
self.x=lat
self.y=long
self.popup=popup
self.f.write(‘L.marker([‘+str(self.x) +’,’+str(self.y)+’]).bindPopup(“‘+str(self.popup)+'”).addTo(map);\n’)
def onclick():

def makeMap(self):
self.f.write(‘</script></body></html>’)
self.f.close()

To use the code, follow the example below.

>>> from pyLeaflet import l
>>> L=l(“Paul.html”)
>>> L.map(35,-106,8)
>>> L.marker(35,-106)
>>> L.marker(34,-106,”Hello from Python”)
>>> L.makeMap()

The output will be an HTML file called paul.html that displays a map with a maker.

Load GeoJSON in ArcMap

2 Feb

It came up on Twitter that you cannot edit GeoJSON in ArcMap. That gave me the idea to try and write a plugin that allows just that. This post will layout a procedure for editing GeoJSON in ArcMap. The final code solution needs work, but it is a good start.

Reading GeoJSON in ArcMap

I grabbed a text file from my Github and saved it to my desktop. This is not the best example, but it was good enough to test the process. I started by reading the file in to ArcMap. The code reads in the file and converts it to JSON. It then iterates through the features, grabbing the coordinates and putting them in to an array.  Lastly, it writes it to a default geodatabase.

import arcpy
from arcpy import env
import json
d=open(r”C:\Users\paul\Desktop\art.geojson”)
data=json.load(d)
points=[]
i=0
for x in data[“features”]:
points.append(arcpy.PointGeometry(arcpy.Point(data[“features”][i][1],data[“features”][i][0])))
i=i+1
arcpy.CopyFeatures_management(points,”testlayer13″)

You will have a map like the image below.

GeoJSON pointsl oaded via Python in ArcMap

GeoJSON pointsl oaded via Python in ArcMap

Writing GeoJSON in ArcMap

You can now use the editing toolbar to draw and edit features as you would any other data source in ArcMap. In the image below, I have deleted most of the features.

Features remaining after edit.

Features remaining after edit.

 

When you have finished editing, you can save out the data as GeoJSON by reading the feature class, grabbing the field that holds the geometry and then writing it out to a text file.

arcpy.env.workspace=r”c:\users\paul\documents\arcgis\default.gdb”
desc=arcpy.Describe(r”c:\users\paul\documents\arcgis\default.gdb\testlayer13″)
shapefieldname=desc.ShapeFieldName
rows = arcpy.SearchCursor(“testlayer13″)
out=open(r”c:\users\paul\desktop\DONE.txt”,”w+”)
out.write(‘{“type”: “FeatureCollection”,”features”: [\n’)
for row in rows:
feat=row.getValue(shapefieldname)
pnt=feat.getPart()
out.write(‘[‘+str(pnt.X)+’,’+str(pnt.Y)+’],’)
out.write(‘]}’)
out.close()

The output is a textfile with GeoJSON.

The GeoJSON output

The GeoJSON output

 

Not pretty, but a start for how I would go about it. Would need to figure out how to do this for polygons and polylines.

Handling Dates in Open Data with JavaScript and Python

13 Jan

Have you ever opened public data and been greeted by a date field with a value like 1419490800000? If it is not obvious to you that the number represents December 25, 2014 then you are not alone. The date has been returned to you in milliseconds since January 1, 1970 at 00:00:00. It might be difficult for you and me to do the math and get a human readable value but for the computer it is a trivial operation. In this article, I will show you how to manipulate a date in milliseconds using data from cabq.gov.   Albuquerque has a growing collection of open data and one newer data sets is Crime Incidents. For this example. let’s grab the REST endpoint. Scrolling to the bottom of the page and clicking the supported operation link QUERY, we can look at the data before coding against it. In the form, set the where field to 1=1, the out fields to * and click the Query (GET) button. It will take a few minutes, but I received 24,629 records (this data set returns incidents for the last 180 days).

Sample of my query results

Sample of my query results

As you can see, the date field is in milliseconds. We will move on to writing a webpage to grab the data and return the date in a variety of other formats – which will be more useful for building our applications.

JavaScript Date

There are many date libraries available in JavaScript but for this example we will only use the built-in date object. The first thing we need to do is grab some data. I will start by using a query to grab a subset of incidents from 1/9/2015 until today. While the data is returned in milliseconds, we can query it using a human readable date format. Below, I have created a variable params by concatenating several strings. The strings use the escape() function because browsers tend to garble URLs with special characters and the function allows us to pre-encode them so we know what the conversion of the characters will be.

q1=’where=’; q2=escape(‘”date”>date’); q3=escape(” ‘2015-01-09 00:00:00′”) ; q4=”&outfields=*&f=json”; var params=q1.concat(q2,q3,q4);

Now that we have the parameter to pass, the rest is a standard AJAX request to the URL of the service.

var url = “http://coagisweb.cabq.gov/…/query “; http=new XMLHttpRequest(); http.open(“POST”, url, true); http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”); http.onreadystatechange = function() { if(http.readyState == 4 && http.status == 200) { //Code goes here }} http.send(params);

Now we have queried the data and have an if statement that will run when the server hands us our results. The first thing to do is grab the response and convert it to JSON. Then we will grab the date from the first result and display it in the console.

theGeom= JSON.parse(http.responseText); adate=new Date(theGeom.features[0].attributes.date); console.log(“RESPONSE FROM ESRI: ” + theGeom.features[0].attributes.date);

If we were successful, you should see a millisecond date in the console. Using the date object, you can convert milliseconds to any of the below formats. Note that months are indexed starting at 0. That means January is month 0 and December is month 11. You need to add 1 to a month for it to be correct.

console.log(“TIME STRING: ” + adate.toTimeString()); //0=january 11 = december console.log(“MONTH: ” + adate.getMonth()+1); console.log(“DAY: ” + adate.getDay()); console.log(“YEAR: ” + adate.getFullYear()); console.log(“HOURS: ” + adate.getHours()); console.log(“MINUTES: ” + adate.getMinutes()); console.log(“SECONDS: ” + adate.getSeconds()); console.log(“ISO 8601: ” + adate.toISOString()); console.log(“UTC: ” + adate.toUTCString()); console.log(“STRING: ” + adate.toString()); console.log(“LOCALE: ” + adate.toLocaleDateString());

Results in the console

Results in the console

Using Python

Using Python, we can do exactly the same thing as the JavaScript example, but this time we will run the code of our desktop. We will need to import the urllibs and datetime. I have chosen to use a third party library simplejson.

import urllib, urllib2, datetime, simplejson

To query a REST endpoint in Python, you will:

  1. create a parameters object
  2. concatenate it to a url
  3. then open the url and read the reply.

param = {‘where’:’1=1′,’outFields’:’*’,’f’:’json’} url = ‘http://coagisweb.cabq.gov/…/MapServer/0/query? ‘ + urllib.urlencode(param) rawreply = urllib2.urlopen(url).read()

If you print rawreply, you will see a large string. You will need to convert the string to JSON. This is where I used simplejson.

reply = simplejson.loads(rawreply) print reply[“features”][0][“attributes”][“date”] print reply[“features”][0][“attributes”][“CVINC_TYPE”]

Your program should have printed out the millisecond date for the first result and the incident type. Now, we can assign the date to a variable and convert it to a readable format.

d=reply[“features”][0][“attributes”][“date”] readable = datetime.datetime.fromtimestamp(d/1000) print readable

The above code should print out something like: 2014-07-14 18:00:00. Now that you have the readable date, you can extract and convert it.

print “YEAR: ” + str(readable.year) print “MONTH: “+ str(readable.month) print “DAY: “+ str(readable.day) print “HOUR: ” + str(readable.hour) print “MINUTE: ” + str(readable.minute) print “SECOND: ” + str(readable.second) print “ISO: ” + str(readable.isoformat())

When all the code is executed, you will see the image below.

Console output of python code

Console output of python code

There you have it, converting milliseconds to a human readable date using two different languages.

MongoDB, pymongo and GridFS

11 Sep

It has been a while since I’ve done anything with MongoDB-I changed jobs and don’t get to code much anymore. I had the urge to learn more and was interested in storing files in MongoDB using GridFS. I googled, read StackOverflow and MongoDB in Action. The problem was most info was storing text as a file or even when storing a file, the code got the _id when executing the put. So of course get is easy, you have the id. Figuring how to get after the fact was where I got stuck, also had to switch file operations to binary. Here is what I have for putting a file in MongoDB and how to retrieve it later.

from pymongo import MongoClient
import gridfs

client=MongoClient()
db=client.mytest
data=open(“image.png”,”rb”)
fs=gridfs.GridFS(db)
thedata=data.read()
stored=fs.put(thedata,filename=”inmongoimage”)

To get it back in the same code you call:
out=fs.get(stored).read()

This works because stored has the _id of the put operation. But what if I need to retrieve in different code from the inserts? Here is how I got it out with some extra code for info:
imports….
client=MongoClient()
db=client.mytest
fs=gridfs.GridFS(db)

filelist=fs.list()
#returns the file names stored

fileone=filelist[0].encode(“ascii”,”ignore”)
#returns a string of the first file name

outdata=fs.get_version(fileone,”rb”).read()
output=open(“somefile.png”,”wb”)
output.write(outdata)
output.close()

Put allows for more metadata than just the filename:
fs.put(thedata,filename=”file.jpg”,field=”string of text”,anumberfield=52)

To find on a different field:
fs.get_version(anumberfield=52)

Python in C# Using IronPython

16 May
C# calling Python

C# calling Python

I am better at Python than C# and when I couldn’t do something in C#, I would use:

System.Diagnostics.Process.Start(“DoItInPython.py”);

I had seen some posts about IronPython but always ignored them – to my own detriment. IronPython is the solution! You can call Python from C# or C# from Python. That means I can use System.Windows.Forms in my Python code.

Reading a post from MSDN, I put together this example that onClick() of button1 it passes the string from the textbox to a python function that just adds “Hello: ” to it. The python function is in name.py:

def printName(name):
return “Hello: “+name

The C# using IronPython:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using Microsoft.Scripting.Hosting;
using IronPython.Hosting;
namespace WindowsFormsApplication1
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}

private void button1_Click(object sender, EventArgs e)
{
var ipy = Python.CreateRuntime();
dynamic test = ipy.UseFile(@”C:\Documents and Settings\user\Desktop\name.py”);
MessageBox.Show(test.printName(textBox1.Text.ToString()));
}
}
}

You can go the other way too. Read the documentation for IronPython.net for more information than I will give here. Here is a Python script using a Windows Form.

Python calling System.Windows.Forms.MessageBox.Show()

Python calling System.Windows.Forms.MessageBox.Show()

You can see it took me a minute to get it working using my own DLL. Here is how to use a System DLL:

import clr

clr.AddReference(“System.Windows.Forms”)

import System.Windows.Forms as f

f.MessageBox.Show(“Hello From Python!”)

This will popup a MessageBox. Simple, right? I have a DLL on my desktop I wrote. As you can see in my screenshot above, I got it to come in almost the same way – just use AddReferenceToFileAndPath(“C:\desktop\yourname.dll”). Here is the full code to call the method MyName() – yourname is the namespace, Name is the class and myName is a method:

import clr

clr.AddReferenceToFileAndPath(“C:\desktop\yourname.dll”)

import yourname as n

a=n.Name()

a.myName(“Paul”)

Ouputs: ‘Hello:  Paul’

Now I can use Python in C# and Windows DLLs in Python. I’m positive this will come in handy.

RabbitMQ, Python and C#

17 Apr
C# consuming a message from python producer via RabbitMQ.

C# consuming a message from python producer via RabbitMQ.

I don’t know much about software patterns. My applications tend to be more of the script variety – though I throw some OOP in for fun. I recently came across RabbitMQ – a message broker. It sits between producers and consumers of messages – it is the queue. I tested out the python hello world example and then got the bright idea to use C# as one piece. The thing about RabbitMQ is that it uses a standard Advanced Messaging Queuing Protocol. This means any language or application that implements that protocol can communicate – think HTTP or FTP protocols.

Here is the python producer taken from the RabbitMQ Tutorial. You will need pika.

import pika
import logging
logging.basicConfig()
connection = pika.BlockingConnection(pika.ConnectionParameters(host=’localhost’))
channel = connection.channel()

channel.queue_declare(queue=’hello’)

channel.basic_publish(exchange=”,routing_key=’hello’,body=’Hello World!’)
print ” [x] Sent ‘Hello World!'”
connection.close()

 

Simple enough. Now I wrote a C# form that consumes the messages onClick of a button and displays them in a text box – appending the new messages as they come in.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using RabbitMQ.Client;

 

namespace WindowsFormsApplication1
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}

private void button1_Click(object sender, EventArgs e)
{
var connectionFactory = new ConnectionFactory();
IConnection connection = connectionFactory.CreateConnection();
IModel channel = connection.CreateModel();
channel.QueueDeclare(“hello”, false, false, false, null);
BasicGetResult result = channel.BasicGet(“hello”, true);
if (result != null)
{
string message = Encoding.UTF8.GetString(result.Body);

textBox1.Text += message+”\r\n”;
}

channel.Close();
connection.Close();
}
}
}