Archive | Programming RSS feed for this section

Graph Databases II: Routing

18 Aug

I wrote a previous post on graph databases that did little more than add some nodes and a relationship. The post required me to install neo4j and python py2neo. It was simple, but it was a start. I have moved on to other things since then, most notably Go. But I have found myself coming back in a round about way. I recently wrote a plugin for Open Trail Data in Leaflet. Trails are a network but the data is really just lines. I thought if we add nodes at the starting points, intersections and eds of the trails, we could use them for routing. I set off to work.

The Data

The first step was grabbing a simple network. I used the Open Trail Data dummy data. I was going to draw nodes then import them in to a graph database, but I decided to shelve that for later – when I know I can route. The data is drawn in the image below.

Open Trail Data

Open Trail Data

The Graph

Next, I needed to create the graph database. I started with all the endpoints of the lines and then added nodes to the intersections. The code below shows how I created two nodes and a path between them.

one = Node(“trail”,id=”1″,junction=”1″)
oneint = Node(“trail”,id=”1″,junction=”1,2,3,4″)
p=Path(one, Rel(“connects”),oneint)

The end result was a graph database as seen in the image below.

Finished Graph Database

Finished Graph Database

Route

Now it was time to route. I do not have weights or costs on the edges. I just wanted to see if I could get from one point to another. I chose node 3 to node 6. They are both on the left. I like it because there are two possibly ways to go. To route, I used the following query:

results=graph.cypher.execute(“START beginning=node(22),end=node(32) MATCH p = shortestPath(beginning-[*..5000]-end) RETURN p”)

The nodes are 22 and 32 because that it the internal ID assigned by the database. My result is a path:

(:trail {id:”3″,junction:”3″})-[:connects]->(:trail {id:”1″,junction:”1,2,3,4″})-[:connects]->(:trail {id:”1″,junction:”1,7″})-
[:connects]->(:trail {id:”7″,junction:”7,6,8,9,12″})-[:connects]->(:trail {id:”6″,junction:”6″})

This tells me the database is going from node 3 to the intersection of 1,2,3,4 to the intersection of 1,7 to the intersection of 7,6,8,9,12 to the end node 6.

You can see the nodes by using

resultPath = results[0][0]
solved=newPath.nodes
for n in solved:
   print n

Now I know all the nodes required to get from one point to another. If I added (lat,long) values to each node, I could have drawn the path on a map.

I will put this in my pocket for later. I am sure it will come up again and I will be better suited to solve a problem. In the meantime, I think I will work on figuring out how to get a standard data format in to the Database – shapefile or geojson?

ESRI REST Geocoding Service in Go

9 Jul

The first thing I wanted to do with Go was to connect to a REST Endpoint. This is an easy task, something I do a lot, and it provides some great possibilities for larger applications. The full code is on GitHub Code, but I will walk through it section by section below.

First, declare the package and import the needed libraries.

package main

import (
“fmt”
“net/http”
“io/ioutil”
“encoding/json”)

We us fmt to print out the result to the console, net/http to connect to the ESRI service, io/util for reading the data the page returns and encoding/json to read the results -which are returned as json.

Next, we need to define the structure of our results.

type geocoderesult struct{
Candidates []candidates
}
type candidates struct{
Address string
Location struct{
X float64
Y float64
}
}

The GeoJSON result from the geocoding service will have a bunch of data and the results are stored in an array named candidates. You need a struct that will grab that result set. The struct geocoderesult has a Candidate []candidates. We need to define candidates. The second struct defines the candidate as contianing an Address and a Location. Location is also a struct that contains and x and y value. The structs match the JSON response.

Now, we create our main function to run the program.

func main(){

response, err := http.Get(“http://server/Geocode/findAddressCandidates?Street=100+lomas&f=json”)
if err != nil {
fmt.Printf(“%s”, err)

}else{}

}

The function connects to the service and passes the parameters street and f. These are part of the ESRI REST API and I have explained them in other posts. The result of http.Get returns a response and an error. If there is an error, we Printf it. In the else statement, we can print the results.

defer response.Body.Close()
c, _ := ioutil.ReadAll(response.Body)
var data geocoderesult

json.Unmarshal(c,&data)
fmt.Println(data.Candidates[0].Address)
fmt.Println(data.Candidates[0].Location)

We close the Body when we are done with it – defer allows us to keep the close close to where we open it. Next, we read the body of the response. In Go, there are a lot of functions that return more than one value (a value and an error). If you do not want to use one of the values returned, you need to use an underscore. Otherwise, if you declare it with a name and don’t use it later, the application will not run.

The next line declares data as a new empty geocoderesult. We then unmarshal(decode) the json response – c – to data. Notice the ampersand before data? Go uses pointers. The ampersand says that we want to put the unmarhaled json data in to the location of data in memory. Basically, replace the current value of data with the json.

Lastly, we can grab the first result returned from the service and print it. Indexes start at 0. Data contains a slice of Candidates so we can index on it using data.candidates[x]. Then candidates has fields for address and location.

Now you can build the application and run it. To make it more useful, you could have it read a textfile of addresses and write out a new file with their locations. Or, reverse geocode a series of points. Then you could build the binary and run it all the time.

In my next post, I will show you how to wrap this code in to a library so we can call it from another program as a function ourlibrary.geocode().

Go Library for ESRI REST

9 Jul

In my last post, I showed how to consume an ESRI REST geocoding service using Go. In this post, I will consume a feature service and query it. This post will take it one step further. We will put the code in a library so that anyone can use it in their applications. The full code is on GitHub.

Usually, the first line is package main, but we will change it to the name for our new package. then we can import the same libraries as in our geocoding sample.

package esrirest

import (
“fmt”
“net/http”
“io/ioutil”
“encoding/json”

)

The structs that we will create will be different as well. In the geoocoding example we wanted candidates. When running a query, we want features.

type query struct{
Features []features
}

ESRI GeoJSON uses Attributes instead of properties, so we will create a map of type interface for the attributes in the feature. This allows us to grab any data types without having to hardcode them as we did in the geocoding example (address, location {x,y}).

type features struct{
Attributes map[string]interface{}
}

In case we need it later – I ever get around to adding more functionality – I will put the variable data outside the function.

var data query

Now, instead of writing a main() function, we will write a function Query – an not include a main in this program because it is a library, and not meant to be executed stand alone. The Query functino will take a string (URL with parameters) that points to the ESRI service. It returns a slice of features. The rest of the code looks exactly like the geocoding example.

func Query(s string) []features {

response, err := http.Get(s)
if err != nil {
fmt.Printf(“%s”, err)

} else {
defer response.Body.Close()
c, _ := ioutil.ReadAll(response.Body)
json.Unmarshal(c,&data)

}
return data.Features
}

The one difference is that the function returns the slice of features – data.Features. You now have a functioning library that you can use in any program.

Use the Library

I will now show you how to use the library.

Run the command:

go get github.com/PaulCrickard/Go/esrirest

This will grab the plugin and install it locally for you. Now write your application.

Declare the package to be main and import the needed libraries. In this case we only need fmt and our new library that we created, downloaded and installed.

package main

import (
“fmt”
“github.com/PaulCrickard/Go/esrirest”

)

If you look in your pkg directory, you will see a folder with your OS type (windows_amd64) then  github.com, PaulCrickard, Go and a file esrirest.a. That is the path you use for the import.

Create your main function and now just call the function in our library passing the URL.

func main(){
//Pass query parameters in URL. MUST HAVE —–> f=json
d :=esrirest.Query(“http://coagisweb.cabq.gov/arcgis/rest/services/public/PublicArt/MapServer/0/query?where=1=1&outFields=*&f=json”)
}

All of the data is now in d. We can now access features and attributes.

fmt.Println(d[0].Attributes[“TITLE”]

Or, we can iterate through all the features and print the titles.

for i:=0; i<len(d); i++{
fmt.Println(d[i].Attributes[“TITLE”])
}

What if you do not know what the attribute names (keys) are? You can iterate through and grab them.

for key, value := range d[0].Attributes{
fmt.Println(key,value)

}

This will print out the key and value for the first feature. Throw this inside your for loop above to go through each feature.

You can grab just the keys by using:

for key, _ := range d[0].Attributes{
fmt.Println(key)

}

Or print the values:

for key, _ := range d[0].Attributes{
fmt.Println(d[0].Attributes[key])

}

Now you know how to crate a library, how to install it and how to use it in another application.

 

The Go Language

9 Jul

I am primarily a Python and JavaScript developer  (I throw in an occasional C# application which is usually for creating REST endpoints on the server side). But recently, I have started looking more seriously at Go.  The Go language is not replacing JavaScript for me, but it could replace a big chunk of my Python and C# code.

I see two big advantages in using Go – besides being fast and the whole concurrency brouhaha:

  1. Cross platform
  2. No dependencies.

Before you yell that Python is cross platform too, let me explain. The same Go code runs on Windows, MAC and Linux. So does Python, right? Well, Go code compiles to binaries for each of the operating systems. This is a big deal for me and one of the reasons I like C#. I can just deploy the .exe and it runs. No command line. No .bat file to pretend it is an .exe. This brings me to the second advantage.

Everything you need to run a Go application is in the binary. C# is great, but I can’t count the number of times I deployed an application and the user had an old version of .NET and it didn’t work. With Python it’s even worse. My development machine has every third party library I need, but getting this setup on every machine I want to deploy to is a nightmare (Docker looks like it could help with this).

There are other things I like about Go – godoc, gofmt, goget. There are also things I don’t like (I am not proficient with)  – pointers.

In later posts, I will share my Go code. For now, here is the obligatory Hello World application.

package main

import “fmt”

func main() {
fmt.Println(“Hello, World”)
}

Python Wrapper for Leaflet

20 Mar

I recently stumbled upon Folium – a python wrapper for leaflet. I was excited and it seemed to work well. I slowly ran in to problems and the pages loaded slow. I probably did something wrong on my end, but decided to write a simple wrapper on my own.

My wrapper is a python function for different Leaflet features such as map and marker. When you call each function, it writes a string to a file to generate the HTML. Below is my python code (pyLeaflet.py).

class l(object):

def __init__(self,path):
self.path=path
self.f=open(self.path,”w+”)
self.f.write(‘<html><head><title>Map From Python</title><link rel=”stylesheet” href=”http://cdn.leafletjs.com/leaflet-0.7.2/leaflet.css&#8221; /></head><body><script src=”http://cdn.leafletjs.com/leaflet-0.7.2/leaflet.js”></script><div style=”height:900px; width:900px” id=”map”></div><script>\n’)

def map(self, lat,long,zoom):
self.lat=lat
self.long=long
self.zoom=zoom
self.f.write(“var map = L.map(‘map’, {center: [“+str(self.lat)+”,”+str(self.long)+”], zoom:”+str(self.zoom)+”});\n”)
self.f.write(“L.tileLayer(‘http://{s}.tile.osm.org/{z}/{x}/{y}.png’).addTo(map);\n”)
def marker(self,lat,long, popup=””):
self.x=lat
self.y=long
self.popup=popup
self.f.write(‘L.marker([‘+str(self.x) +’,’+str(self.y)+’]).bindPopup(“‘+str(self.popup)+'”).addTo(map);\n’)
def onclick():

def makeMap(self):
self.f.write(‘</script></body></html>’)
self.f.close()

To use the code, follow the example below.

>>> from pyLeaflet import l
>>> L=l(“Paul.html”)
>>> L.map(35,-106,8)
>>> L.marker(35,-106)
>>> L.marker(34,-106,”Hello from Python”)
>>> L.makeMap()

The output will be an HTML file called paul.html that displays a map with a maker.

Albuquerque Open Data

18 Mar

I have put together several posts on Albuquerque Open Data and have felt bad that the examples were not live. I shutdown my old website and tried to use JsBin for a few examples, but I was not consistent and couldn’t upload third party JavaScript files. Well, I found a solution.

GitHub Pages to the Rescue

I have a GitHub repo. A GitHub Repo is a place to put code where you can harness the full power of git – branching, committing and others can pull and push code to your repo. GitHub also has a thing called GitHub Pages where you can host sites. I have one. And I am now using it to host my presentations. I do not own Albuquerque Open Data, but the community does. The City puts the data out there for us to use. As a community member, I decided to create a GitHub repo for Albuquerque Data and a GitHub page for it. You can download the code for all the examples, or run them. You can also grab a copy and modify thm. Lastly, you can add your own code examples to the repo.

The Examples

There are only a few examples thus far.

  • Aerial – Zoom to a location in ABQ. Double click to get back an aerial image of the map window.
  • Crime – Click the map to see incidents near you.
  • Density
  • Elevation – Draw a line with lots of segments to see an elevation profile.
  • Game – Click the map to add a marker. Drag the marker to a park to win.

The Code for the Examples

You can get the code for each example by clicking the link.

  1. Aerial
  2. Crime
  3. Density
  4. Elevation
  5. Game

You can also grab the whole Repo as a ZIP file.

Reverse Geocoding a Line

3 Mar

In Geocoding with ABQ Open Data, I showed you how to geocode an address and how to reverse geocode a point when the user clicks on the map. Geocoding is an operation performed on points or addresses, so what do we do with lines? In this post I will show you how I am geocoding a line.

The Need

I have an application that allows a user to select a location for where they would like to perform some construction work and need a permit. The application started by allowing the user to specify a point or address. As I met with the users of the application, it turned out that having line features was crucial. Work may be performed along a stretch of road and the approvers of the permit wanted to be able to find at least the starting address when looking at the data in a tabular format – mostly for things like how many permits are on Central Ave.

The Solution

Alert box showing the start and end addresses of a line

Alert box showing the start and end addresses of a line

The solution uses the same reverse geocoding process as my previous post, but parses a line feature in to points. I am using the leaflet.draw plugin to allow the user to draw the line on a map. The draw plugin has a draw:created event that searches to see the type of object drawn. My code runs in the if (type == ‘polyline’) block.  The code below converts the layer drawn to GeoJSON and then the GeoJSON coordinates to a string split on the comma. Using the string at index 0 and 1, I have the starting point of the line. Using the index of length-1 and length-2 I can grab the last point. I use length because the user can draw a line segment with an unlimited number of points so I will not know where the line ends and do not want to assume a two point line. when I have the points, I call reverseGeocode().

var tojson=layer.toGeoJSON();
var x=String(tojson.geometry.coordinates).split(“,”);

//start point
var lt=parseFloat(x[1]);
var ln=parseFloat(x[0]);
reverseGeocodeLineFrom(ln,lt);

//endpoint
var ltend=parseFloat(x[x.length-1]);
var lnend=parseFloat(x[x.length-2]);
reverseGeocodeLineTo(lnend,ltend);

The reverseGeocode function just makes an AJAX call to the ESRI REST API service for my geocoder passing the point and parsing the response.

function reverseGeocodeLineFrom(a,b){

var urlgeocode=”http://servername/ArcGIS/rest/services/wgs84locator/GeocodeServer/reverseGeocode&#8221;;

var geocodeparams = “location=”+a+”,”+b+”&distance=1000&f=json”;
var http;
http=new XMLHttpRequest();
http.open(“POST”, urlgeocode, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {//Call a function when the state changes.
if(http.readyState == 4 && http.status == 200) {

addressAsJSON=JSON.parse(http.responseText);
alert(addressAsJSON.address.Street);
}
}
http.send(geocodeparams);
}

Now I have a street address for the start and end of my line.