Tag Archives: Open Data

Data and Design

8 May

If you have read any of the posts on this blog, you should know I love data. But what you may not know is that I love architecture, design and a good sketch. I spent five hours getting Study For The War Coffer by Eugene Delacroix tattooed on my chest.

coffer

Often these two worlds collide. I came across a tweet today:

Mindlessly drawing with data? How dare she. I once thought it a good idea to write computer code that could read an architectural program and develop the floor plan automatically. While I still favor some of this thinking, I have had to think it through. And slowly I have come out against it, and I have sided with Tara on this issue.

There is something to be said for hand drawing. The lines made by a pen, with their varying weights, show movement in a still image. There is something beautiful about them. About the process of sketching. Freely moving your hand across the canvas. The AIA had a podcast on Didactic Drawing that really brought it all home for me. On a computer, scale can change. You can draw a hundred foot line and based on your zoom level (scale) it could be a millimeter long. On paper, your scale is fixed. The movement of your hand across the page lets you know how long the line is.

I am not against BIM. But without pre-sketching designs, these program make it easy to create boxes, squares and overall bland buildings, to draw without a set scale, to fully understand and feel the building you are creating.  To design with data is an idea I am still deeply attached to. But I think we walk a fine line between letting data inform design- on how people use buildings for example – to creating the design for us – as in my program example earlier.

Applications like Revit or Grasshopper make it east to start with a simple form – a box – and twist, pull, rotate and skew it to come up with a whole host of possible forms. The results are soulless – though some look really cool. I do not see the art in it. If we are just going to feed some data in to a model to generate a form and say “look at this cool form I created from using the coordinates of all tweets that had the word Gehry in it” then we might as well give up – though I find these kinds of experiments interesting.

Data is, of course, valuable for facility maintenance. I also find value in data on movements of individuals within buildings and with modeling designs for things like airflow, heat, sunlight, etc. These are the kinds of data that can inform design – or confirm that a specific design is a functional design.

I do not want to live in a City full of bland buildings, just as much as I do not want to live in a world full of monuments to the architect that are outrageously out of context. There needs to exist a balance of the art and the science, of architecture and data. And each needs to compliment the other.

 

 

Advertisements

Graph Database and Albuquerque Bus Stops: Neo4j with py2neo

15 Apr

I have been slightly obsessed with the question: “How do you define network service areas client-side on a map.” I know it needs a networked data set and something to do with the Djikstra algorithm (Yes, we could just use an ESRI REST service but there is not one available yet – I will ask the City). After looking at JavaScript implementations of NetworkX, I stumbled upon graph databases, most notably Neo4J.  A networked data set is a graph. Guess what, it has Djikstra built-in, so I must be on the right path. I installed it and added a fake social graph using py2neo. That allowed me to make sure I could do a few things:

  • Add a node
  • Add a relationship
  • add attributes

Now it was time to start with some real data.

My first test was to load Albuquerque Bus Stops for a single route. Here is what I have in my database.

Bus Stops for Route 766. No Relations added yet.

Bus Stops for Route 766. No Relations added yet.

The image above was generated by calling the City of Albuquerque REST Endpoint for bus stops, parsing the response, and putting it in to Neo4J. The image is a view from the DB Manager. The code to do this is below.

from py2neo import Graph
from py2neo import Node, Relationship
from py2neo import authenticate
import urllib2
import json

authenticate(“localhost:7474″,”myUserName”,”myPassword”)
graph=Graph()
graph.delete_all()

url=”http://coagisweb.cabq.gov/arcgis/rest/services/public/fullviewer/mapserver/22/query?where=ROUTE=’766’&f=json&outFields=*&outSR=4326″
rawreply=urllib2.urlopen(url).read()

reply=json.loads(rawreply)

for x in reply[“features”]:
graph.create(Node(“stop”,route=x[“attributes”][“ROUTE”],direction=x[“attributes”][“DIRECTION”],street=x[“attributes”][“STREET”],intersection=x[“attributes”][“NEAR_INTER”],lat=x[“geometry”][“y”],long=x[“geometry”][“x”]))

Notice there are no Relationships! This is crucial if we will ever walk the network. I have manually added on, seen in the image below.

San Mateo links to Louisianna.

San Mateo links to Louisianna.

The code for this is:

rel=Relationship(graph.node(42),”Next”,graph.node(41))

graph.create(rel)

I need to think about how to automate the relationship creation based on stop order and direction (there are stops on both sides of the street). Then, I will need to figure out how to make a node have relationships to other routes. For example, many stops are connected to the 777 route and I do not want a separate node for each. I want one with a property showing routes.

Well, a start to say the least. It has been fun learning about graph databases and if GIS doesn’t interest you, you could map your social network and walk it.

Open Data Disclaimers and Terms of Service

7 Apr

I saw a tweet by @waldojaquith that commented on the State of NY Open Data Portals TOS. The TOS states that you cannot access the page

“…by using an automated device, script, bot, spider, crawler or scraper”

I can guess their intent is to prevent people from hitting the site with bots and overloading the server. But a device and script? This seems at best, too broad, at worst, ignorance as to how people will want to use the data. Furthermore, will I be prosecuted from doing so or will my IP address be blocked? Is this even enforceable? Can a government block a private citizen from a civic website?

This tweet made me curious about the city I live in – Albuquerque. I was dumbfounded by the content of their disclaimer/TOS. It read

“The City may require a user of this data to terminate any and all display, distribution or other use of any or all of the data provided at this website for any reason…”

Really? The City can call me and tell me to remove the bar chart of car thefts by month from my website because it is based on their data? Or, that I can no longer send the PublicArt.kmz file to someone?

I love Albuquerque Open Data. I find it quite useful and very good. So when the page says that

“The City makes no warranty, representation, or guaranty as to the content, accuracy, timeliness, or completeness of any of the data provided at this website.”

it makes me question the quality of the data. I get it, the data might be wrong, don’t hold it as gospel. Seems to be common sense to me. But this statement creates a distrust in the data. Is it authoritative? After reading the TOS, I would not take it to be.

If I were a gambling man, I would bet this is the work of the Legal Department. The same people that probably have email signatures saying if they send you confidential information by mistake you are in trouble for not deleting it. Yeah, I don’t think so Matlock. Saying it doesn’t make it true.

Government is made up of many departments. Those departments create a bunch of data. That data finds its way to an open data website probably run by the IT Department. The IT department just puts it out there and doesn’t really know anything about the data. They don’t make it so why would they stand by it? They point you to the department that made it. But that department probably didn’t care to release it in the first place and don’t want to be bothered exporting it or updating it. Or worse yet, fielding calls from citizens about it. But if someone is going to put out some data, there needs to be a level of trust or quality in the data.

Is it better to put out some data of quality by choice than to be legally obligated to provide it as the result of a FOIA request? I would imagine open data cuts down on a lot of those requests, saving a lot of time in money.

Once you put it out, please don’t think you have any control over what I do with it or who I send it to.

I do not know the answer so I am putting the question out there, and I did so on Twitter as well:

Has a City ever been sued for the quality of their open data?

 

Historic Bus Location Data

27 Mar

Last night I was trying to think of interesting uses for a Rasberry Pi. One thing I came up with was a data logger. But what to log? Then I thought about a previous post on the Albuquerque Realtime Bus Data. Hmmm. What if I wanted to show the bus locations using a time slider. What if I want to see if they ever deviated from their routes, or if they deviated from their schedules? I can’t really do any analysis without the historic data, and the City does not give that out currently. So I think I find a use – logging Albuquerque Bus Data.

I don’t have a Rasberry Pi, yet, so I wrote a python script on my desktop to test the logger.

I am not going to post the code because I don’t know the impact on the Albuquerque server. I will give a brief explanation. The City has KML files for each bus route. Each route has multiple buses. I grabbed a single route – 766, and parsed the results. I initially sent the results to a csv – as you can see in the data below this post. Writing to CSV is not too helpful when the data gets large (I am not going to say BIG). Once I knew it worked, I sent the data to a MongoDB that was spatially indexed. In the database, I can now:

Get the total records

                         collection.count()

Get all the Records

                        for x in collection.find():
                                      print x

Get a specific bus number

                       for x in collection.find({‘number’:’6903′}):
                                           print x

Or find near a lat,lng

                       for x in collection.find({“loc”:{“$near”:[35.10341,-106.56711]}}).limit(3):
                                           repr(x)

With a database, multiple people can query it and perform operations on it. Lastly, if the data gets larger, Mongo can be split (sharding) across multiple machines to hold it all.

My MongoDB records look like:

{u’loc’: [35.08156, -106.6287], u’nextstop’: u’Central @ Cornell scheduled at 3:45 PM’, u’number’: u’6903′, u’time’: u’3:46:52 PM’,
u’_id’: ObjectId(‘5515cfd814cd2829e4c1b718′), u’speed’: u’20.5 MPH’}

Here is the results of my original run. I ran the script for 7 minutes and got the following results for Route 766.

Bus

Hard to see, but displays bus locations along route 766 over a 7 minute period.

 

6409,0.0 MPH,1:34:02 PM,Central @ San Mateo (Rapid) scheduled at 1:31 PM,-106.58642,35.07778
6904,1.9 MPH,1:34:03 PM,Central @ Edith scheduled at 1:31 PM,-106.64776,35.08401
6411,0.0 MPH,1:33:56 PM,CUTC Bay B scheduled at 1:39 PM,-106.72266,35.07726
6407,21.7 MPH,1:34:00 PM,Central @ 1st (across from A.T.C.) scheduled at 1:34 PM,-106.64317,35.08359
6410,35.4 MPH,1:33:53 PM,Central @ San Mateo (Rapid) scheduled at 1:36 PM,-106.58247,35.07749
6403,0.0 MPH,1:34:02 PM,Indian School @ Louisiana scheduled at 1:40 PM,-106.57089,35.10305
6903,29.8 MPH,1:33:56 PM,Central @ Atrisco scheduled at 1:36 PM,-106.69049,35.08443
6409,0.0 MPH,1:35:14 PM,Louisiana @ Central (Rapid) scheduled at 1:36 PM,-106.5852,35.07764
6904,1.9 MPH,1:35:14 PM,Central @ Edith scheduled at 1:31 PM,-106.6478,35.08413
6411,0.0 MPH,1:35:07 PM,Next stop is CUTC Bay B scheduled at 1:44 PM,-106.72525,35.07886
6407,0.0 MPH,1:35:11 PM,Copper @ 2nd scheduled at 1:34 PM,-106.64784,35.08417
6410,19.3 MPH,1:35:17 PM,Central @ Carlisle (Rapid) scheduled at 1:40 PM,-106.58734,35.07802
6403,0.0 MPH,1:35:14 PM,Indian School @ Louisiana scheduled at 1:40 PM,-106.57087,35.10306
6903,14.9 MPH,1:35:08 PM,Central @ Tingley (Rapid) scheduled at 1:38 PM,-106.68485,35.08576
6409,23.6 MPH,1:36:38 PM,Louisiana @ Central (Rapid) scheduled at 1:36 PM,-106.58084,35.07723
6904,18.0 MPH,1:35:53 PM,Central @ Edith scheduled at 1:31 PM,-106.647,35.08373
6411,0.0 MPH,1:36:31 PM,Next stop is CUTC Bay B scheduled at 1:44 PM,-106.72525,35.07885
6407,0.6 MPH,1:36:35 PM,Copper @ 5th scheduled at 1:35 PM,-106.64944,35.08541
6410,0.0 MPH,1:36:40 PM,Central @ Carlisle (Rapid) scheduled at 1:40 PM,-106.59512,35.07883
6403,0.0 MPH,1:36:42 PM,Indian School @ Louisiana scheduled at 1:40 PM,-106.57086,35.10306
6903,31.7 MPH,1:36:33 PM,Central @ Rio Grande (Rapid) scheduled at 1:40 PM,-106.67824,35.09252
6409,37.9 MPH,1:37:49 PM,Louisiana @ Central (Rapid) scheduled at 1:36 PM,-106.57203,35.07627
6904,0.6 MPH,1:37:55 PM,Central @ Cedar (Rapid) scheduled at 1:33 PM,-106.63771,35.08276
6411,0.0 MPH,1:37:55 PM,Central @ Coors scheduled at 1:47 PM,-106.72526,35.07885
6407,1.9 MPH,1:37:47 PM,Copper @ 5th scheduled at 1:35 PM,-106.6496,35.08487
6410,0.0 MPH,1:37:52 PM,Central @ Yale (UNM) scheduled at 1:44 PM,-106.60369,35.07979
6403,0.0 MPH,1:37:57 PM,Indian School @ Louisiana scheduled at 1:40 PM,-106.57087,35.10306
6903,0.0 MPH,1:37:56 PM,Central @ Rio Grande (Rapid) scheduled at 1:40 PM,-106.67159,35.09515
6409,0.0 MPH,1:39:14 PM,Louisiana @ Central (Rapid) scheduled at 1:36 PM,-106.56872,35.07595
6904,18.0 MPH,1:38:19 PM,Central @ Cedar (Rapid) scheduled at 1:33 PM,-106.63713,35.08265
6411,0.0 MPH,1:39:19 PM,Central @ Coors scheduled at 1:47 PM,-106.72527,35.07884
6407,3.1 MPH,1:39:10 PM,Central @ Rio Grande (Rapid) scheduled at 1:40 PM,-106.65295,35.086
6410,31.1 MPH,1:39:16 PM,Central @ Yale (UNM) scheduled at 1:44 PM,-106.61167,35.08084
6403,5.6 MPH,1:39:23 PM,Indian School @ Louisiana scheduled at 1:40 PM,-106.5707,35.10368
6903,23.0 MPH,1:39:20 PM,Gold @ 5th (Rapid) scheduled at 1:44 PM,-106.67011,35.09438
6409,26.1 MPH,1:40:37 PM,Louisiana @ Lomas scheduled at 1:38 PM,-106.56848,35.07723
6904,24.9 MPH,1:40:10 PM,Central @ Cedar (Rapid) scheduled at 1:33 PM,-106.63585,35.08255
6411,0.0 MPH,1:40:31 PM,Central @ Coors scheduled at 1:47 PM,-106.72526,35.07884
6407,23.0 MPH,1:40:36 PM,Central @ Rio Grande (Rapid) scheduled at 1:40 PM,-106.65675,35.0863
6410,34.2 MPH,1:40:40 PM,Central @ Yale (UNM) scheduled at 1:44 PM,-106.61567,35.0811
6403,0.0 MPH,1:40:42 PM,Indian School @ Louisiana scheduled at 1:40 PM,-106.56875,35.10221
6903,23.6 MPH,1:40:32 PM,Gold @ 5th (Rapid) scheduled at 1:44 PM,-106.66327,35.08892
6409,0.6 MPH,1:41:49 PM,Indian School @ Uptown Loop Road scheduled at 1:42 PM,-106.56849,35.08691
6904,24.9 MPH,1:41:55 PM,Central @ Cedar (Rapid) scheduled at 1:33 PM,-106.63585,35.08255
6411,0.0 MPH,1:41:55 PM,Central @ Coors scheduled at 1:47 PM,-106.72526,35.07884
6407,0.0 MPH,1:41:58 PM,Central @ Rio Grande (Rapid) scheduled at 1:40 PM,-106.65822,35.08648
6410,28.6 MPH,1:41:52 PM,Central @ Yale (UNM) scheduled at 1:44 PM,-106.6216,35.08113
6403,0.0 MPH,1:42:01 PM,Louisiana @ Lomas scheduled at 1:45 PM,-106.56779,35.10184
6903,23.0 MPH,1:41:55 PM,Gold @ 5th (Rapid) scheduled at 1:44 PM,-106.65819,35.08629
6409,41.0 MPH,1:43:13 PM,Indian School @ Uptown Loop Road scheduled at 1:42 PM,-106.56863,35.09282
6904,24.9 MPH,1:42:15 PM,Central @ Cedar (Rapid) scheduled at 1:33 PM,-106.6217,35.08093
6411,0.0 MPH,1:43:19 PM,Central @ Coors scheduled at 1:47 PM,-106.72528,35.07883
6407,9.9 MPH,1:43:10 PM,Central @ Rio Grande (Rapid) scheduled at 1:40 PM,-106.661,35.08784
6410,34.2 MPH,1:43:16 PM,Central @ Mulberry (Rapid) scheduled at 1:47 PM,-106.62524,35.08125
6403,16.8 MPH,1:43:22 PM,Louisiana @ Lomas scheduled at 1:45 PM,-106.56635,35.10156
6903,19.3 MPH,1:43:19 PM,Gold @ 5th (Rapid) scheduled at 1:44 PM,-106.6549,35.084

Linq in JavaScript

4 Mar

If you are familiar with .NET, then you may know what Linq is: Language Integrated Query. It is a way to query databases or objects. If you have read my blog posts, you will notice that I like to iterate through JSON with if something equal something statements to filter and query my data. In this tutorial, I will show you how to use linq for JavaScript to make queries easier.

The Application

I am going to use Linq.js to process the results of a query to an ESRI REST endpoint.

Capture

Nothing much to see in the image above, just three points with their location type as a popup. It is the logic behind them that is useful. These three points were filtered out from a total of 844. I used the Albuquerque Public Art Data for this example.

The Logic

The map is a standard Leaflet.js map. I made a query to the Albuquerque Public Art endpoint using where set to 1=1. This gives back all the data. I know we can query here, but let’s assume that we are going to provide buttons for the user to filter data and we don’t want to keep querying the service. We grab once and can filter multiple times on any field.

The AJAX query is standard and returns our result.

var url=”http://coagisweb.cabq.gov/arcgis/rest/services/public/PublicArt/MapServer/0/query”;
var params = ‘where=1=1&f=json&outFields=*&outSR=4326’;
var http;
http=new XMLHttpRequest();
http.open(“POST”, url, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {//Call a function when the state changes.
if(http.readyState == 4 && http.status == 200) {
result=JSON.parse(http.responseText);
}}
http.send(params);

We can use linq.js after we convert the response to JSON.

var r = Enumerable.From(result.features)
.Where(function (x) { return x.attributes.OBJECTID > 202261 })
.OrderBy(function (x) { return x.attributes.LOCATION })
.Select(function (x) { return x.attributes.OBJECTID + ‘,’ + x.attributes.LOCATION +’,’+ x.geometry.x + ‘,’ + x.geometry.y })
.ToArray();

The above code does several things.

First, we create the enumerable object from the JSON. When using ESRI JSON, you need to skip to the features or it will rarely parse. Next we select our where clause. The json has two properties that contain the rest (nested): attributes and geography. We take x from the function and then look for items with the attribute for OBJECTID greater then 202261. This gives me three results.

I then order the results by location – for no real reason other than to show it is possible. Then using select, I format the string that will be placed in the returned toArray(). The result is:

[“202263,Academy Hills Park,-106.52671900037603,35.149523000339606”, “202264,Alamosa Park,-106.70765700015008,35.07113600016852”, “202262,City/County Government Building,-106.65203299975377,35.088040999684026”]

Now I have a subset of my data. I can map it:

 for(x=0;x<r.length;x++){
data=r[x].split(“,”);
L.marker([data[3],data[2]]).addTo(map).bindPopup(data[1]);
}

If you want to allow a user to play with some data, grab it all and use linq.js to filter, order and select subsets and only call the server once.

Reverse Engineer City of Albuquerque Web Server Logs API

25 Feb

I recently stumbled across a Code For America experiment that shows live information from the City of Albuquerque Web Server. This is cool, but if you read this blog, you know I want the service! Give me that REST endpoint. I set out to reverse engineering their application and will show you two real-time endpoints with some different parameters.

The Application

The Code for America application is clearly hitting an endpoint. It’s just the way it has to work. So where is the endpoint? Well, a cursory glance at the code reveals two: realtime and historic. So hit the endpoint

https://shrouded-fjord-5635.herokuapp.com/realtime?

and you get the following error:

{“error”:{“errors”:[{“domain”:”global”,”reason”:”required”,”message”:”Required parameter: ids”,”locationType”:”parameter”,”location”:”ids”}],”code”:400,”message”:”Required parameter: ids”}}

Interesting. we need a parameter ids. Hit the endpoint with the parameter.

{“error”:{“errors”:[{“domain”:”global”,”reason”:”invalidParameter”,”message”:”Invalid value ”. Values must match the following regular expression: ‘ga:[0-9]+'”,”locationType”:”parameter”,”location”:”ids”}],”code”:400,”message”:”Invalid value ”. Values must match the following regular expression: ‘ga:[0-9]+'”}}

Now I know the value of

ids=ga: SOME NUMBER

The number is in the code, so I grabbed it. Reading the code, you can see the structure of the endpoint.

endpoint: function(){
return “/realtime?ids=ga:”+matrix.settings.profileId+”&metrics=rt:activeUsers&max-results=10”
}

So let’s give it a try.

{
“kind”: “analytics#realtimeData”,
“id”: “https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&metrics=rt:activeUsers&#8221;,
“query”: {
“ids”: “ga:48754”,
“metrics”: [
“rt:activeUsers”
],
“max-results”: 1000
},
“totalResults”: 1,
“selfLink”: “https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&metrics=rt:activeUsers&#8221;,
“profileInfo”: {
“profileId”: “48754”,
“accountId”: “86004”,
“webPropertyId”: “UA-86004-1”,
“internalWebPropertyId”: “97739”,
“profileName”: “2. http://www.cabq.gov All Traffic”,
“tableId”: “realtime:48754”
},
“columnHeaders”: [
{
“name”: “rt:activeUsers”,
“columnType”: “METRIC”,
“dataType”: “INTEGER”
}
],
“totalsForAllResults”: {
“rt:activeUsers”: “172”
},
“rows”: [
[
“172”
]
]
}

There you have it: 172 active users. We can pass another parameter

metrics=rt:pageViews

Then we see:

{“kind”:”analytics#realtimeData”,”id”:”https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&metrics=rt:pageviews&#8221;,”query”:{“ids”:”ga:48754″,”metrics”:[“rt:pageviews”],”max-results”:1000},”totalResults”:1,”selfLink”:”https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&metrics=rt:pageviews&#8221;,”profileInfo”:{“profileId”:”48754″,”accountId”:”86004″,”webPropertyId”:”UA-86004-1″,”internalWebPropertyId”:”97739″,”profileName”:”2. http://www.cabq.gov All Traffic”,”tableId”:”realtime:48754″},”columnHeaders”:[{“name”:”rt:pageviews”,”columnType”:”METRIC”,”dataType”:”INTEGER”}],”totalsForAllResults”:{“rt:pageviews”:”2172″},“rows”:[[“2172”]]}

I formatted the first result nicely and will just post JSON blobs so as not to take up to much space. Now you see 2172 views. My guess is that this is within either today or some time time frame of today.

What are people looking at? Lets change our metrics to pageviews and get the pagetitle. We also pass the largest possible result value.

metrics=rt:pageviews&dimensions=rt:pageTitle&max-results=10000

Right now, this is what we get.

{“kind”:”analytics#realtimeData”,”id”:”https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&dimensions=rt:pageTitle&metrics=rt:pageviews&max-results=10000&#8243;,”query”:{“ids”:”ga:48754″,”dimensions”:”rt:pageTitle”,”metrics”:[“rt:pageviews”],”max-results”:10000},”totalResults”:454,”selfLink”:”https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&dimensions=rt:pageTitle&metrics=rt:pageviews&max-results=10000&#8243;,”profileInfo”:{“profileId”:”48754″,”accountId”:”86004″,”webPropertyId”:”UA-86004-1″,”internalWebPropertyId”:”97739″,”profileName”:”2. http://www.cabq.gov All Traffic”,”tableId”:”realtime:48754″},”columnHeaders”:[{“name”:”rt:pageTitle”,”columnType”:”DIMENSION”,”dataType”:”STRING”},{“name”:”rt:pageviews”,”columnType”:”METRIC”,”dataType”:”INTEGER”}],”totalsForAllResults”:{“rt:pageviews”:”2127″},”rows”:[[“2015 Spring Green Waste Pick-up — City of Albuquerque”,”1″],[“4th Magic Treehouse: Dinosaurs after Dark — City of Albuquerque”,”1″],[“A Night in the 40’s: Big Band Swing — City of Albuquerque”,”1″],[“ABQ BioPark — City of Albuquerque”,”12″],[“ABQ RIDE Providing Free Transportation to Veterans with V.A. Hospital Cards — City of Albuquerque”,”5″],[“ABQ Recycles — City of Albuquerque”,”2″],[“ABQ View — City of Albuquerque”,”24″],[“ABQ Volunteers — City of Albuquerque”,”6″],[“ACH Services Information — City of Albuquerque”,”1″],[“About 311 — City of Albuquerque”,”1″],[“About Us — City of Albuquerque”,”2″],[“About the BioPark — City of Albuquerque”,”1″],[“About the Councilor — City of Albuquerque”,”1″],[“Activities & Education Programs — City of Albuquerque”,”1″],[“Activities Catalog — City of Albuquerque”,”1″],[“Address Atlas Pages — City of Albuquerque”,”2″],[“Address Query – Advanced Map Viewer — City of Albuquerque”,”5″],[“Address Report — City of Albuquerque”,”19″],[“Administration, Two AFSCME Unions Reach Agreement — City of Albuquerque”,”1″],[“Admission & Hours — City of Albuquerque”,”5″],[“Advanced Map Viewer User Guide — City of Albuquerque”,”1″],[“Advanced Map Viewer — City of Albuquerque”,”14″],[“Agendas from Greater Albuquerque Bicycling Advisory Committee Meetings — City of Albuquerque”,”1″],

Notice there were only 454 total results. Each record looks like

[“ABQ RIDE Providing Free Transportation to Veterans with V.A. Hospital Cards — City of Albuquerque”,”5″]

This is pageTitle comma PageViews as stated in the columnHeaders property below.

columnHeaders”:[{“name”:”rt:pageTitle”,”columnType”:”DIMENSION”,”dataType”:”STRING”},{“name”:”rt:pageviews”,”columnType”:”METRIC”,”dataType”:”INTEGER”}],

Where is the page with the title ABQ View – City of Albuquerque? Using pagePath we can find out.

{“kind”:”analytics#realtimeData”,”id”:”https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&dimensions=rt:pagePath,rt:pageTitle&metrics=rt:pageviews&max-results=10000&#8243;,”query”:{“ids”:”ga:48754″,”dimensions”:”rt:pagePath,rt:pageTitle”,”metrics”:[“rt:pageviews”],”max-results”:10000},”totalResults”:704,”selfLink”:”https://www.googleapis.com/analytics/v3/data/realtime?ids=ga:48754&dimensions=rt:pagePath,rt:pageTitle&metrics=rt:pageviews&max-results=10000&#8243;,”profileInfo”:{“profileId”:”48754″,”accountId”:”86004″,”webPropertyId”:”UA-86004-1″,”internalWebPropertyId”:”97739″,”profileName”:”2. http://www.cabq.gov All Traffic”,”tableId”:”realtime:48754″},”columnHeaders”:[{“name”:”rt:pagePath”,”columnType”:”DIMENSION”,”dataType”:”STRING”},{“name”:”rt:pageTitle”,”columnType”:”DIMENSION”,”dataType”:”STRING”},{“name”:”rt:pageviews”,”columnType”:”METRIC”,”dataType”:”INTEGER”}],”totalsForAllResults”:{“rt:pageviews”:”2173″},”rows”:[[“/311/311-Information/about-311″,”About 311 — City of Albuquerque”,”1″],[“/311/311-Information/contacting-311″,”How to Contact 311 — City of Albuquerque”,”1″],[“/311/311-Information/contacting-311/dialing-311/index.html”,”Dialing 311 — City of Albuquerque”,”1″],[“/311/resident-services”,”Online Resident Services — City of Albuquerque”,”1″],[“/311/resident-services/online-resident-services/folder_summary_view?b_start:int=15&-C=”,”Online Resident Services — City of Albuquerque”,”1″],[“/311/resident-services/online-resident-services/folder_summary_view?b_start:int=30&-C=”,”Online Resident Services — City of Albuquerque”,”1″],[“/a-z”,”Albuquerque A-Z — City of Albuquerque”,”27″],[“/a-z/a-z”,”Albuquerque A-Z — City of Albuquerque”,”7″],[“/about/offsite.html”,”Leaving http://www.cabq.gov External Link Disclaimer — City of Albuquerque”,”1″],[“/abq-apps”,”ABQ Apps — City of Albuquerque”,”1″],[“/abq-data”,”ABQ Data — City of Albuquerque”,”1″],[“/abq-view/index.html”,”ABQ View — City of Albuquerque”,”27″],[“/abq-volunteers”,”ABQ Volunteers — City of Albuquerque”,”9″],[“/abq-volunteers/index.html”,”ABQ Volunteers — City of Albuquerque”,”1″],[“/acl_users/credentials_cookie_auth/require_login?came_from=https://www.cabq.gov/abq-volunteers/volunteering-announcement-form”,”City of Albuquerque”,”1″],[“/airport”,”Airport — City of Albuquerque”,”7″],[“/airport/airlines-flight-services”,”Airlines — City of Albuquerque”,”7″]

Here is a single record:

[“/311/311-Information/about-311″,”About 311 — City of Albuquerque”,”1″]

And appending the URL to cabq.gov gives us:

Capture

 

Now you can see what people are looking at on the City of Albuquerque website. Try gathering statistics and compile the most popular page for a day as Code for America has done.

I said I was only going to show the real-time but I had to grab historic.

Historic Data

You can grab historic data by changing the endpoint to /historic. The metric values change to:

start-date

end-date

They are formatted as YYYY-MM-DD. You can also pass dimensions of:

nthMinute

nthHour

And here is an example of a session from 2015-02-21 to 2015-02-23 using Hours:

https://shrouded-fjord-5635.herokuapp.com//historic?ids=ga:48754&dimensions=ga:nthHour&metrics=ga:sessions&start-date=2015-02-21&end-date=2015-02-23

{“kind”:”analytics#gaData”,”id”:”https://www.googleapis.com/analytics/v3/data/ga?ids=ga:48754&dimensions=ga:nthHour&metrics=ga:sessions&start-date=2015-02-21&end-date=2015-02-23&#8243;,”query”:{“start-date”:”2015-02-21″,”end-date”:”2015-02-23″,”ids”:”ga:48754″,”dimensions”:”ga:nthHour”,”metrics”:[“ga:sessions”],”start-index”:1,”max-results”:1000},”itemsPerPage”:1000,”totalResults”:72,”selfLink”:”https://www.googleapis.com/analytics/v3/data/ga?ids=ga:48754&dimensions=ga:nthHour&metrics=ga:sessions&start-date=2015-02-21&end-date=2015-02-23&#8243;,”profileInfo”:{“profileId”:”48754″,”accountId”:”86004″,”webPropertyId”:”UA-86004-1″,”internalWebPropertyId”:”97739″,”profileName”:”2. http://www.cabq.gov All Traffic”,”tableId”:”ga:48754″},”containsSampledData”:false,”columnHeaders”:[{“name”:”ga:nthHour”,”columnType”:”DIMENSION”,”dataType”:”STRING”},{“name”:”ga:sessions”,”columnType”:”METRIC”,”dataType”:”INTEGER”}],”totalsForAllResults”:{“ga:sessions”:”48097″},“rows”:[[“000000″,”242”],[“000001″,”110”],[“000002″,”91”],[“000003″,”91”],[“000004″,”106”],[“000005″,”147”],[“000006″,”308”],[“000007″,”671”],[“000008″,”912”],[“000009″,”1098”],[“000010″,”1119”],[“000011″,”1019”],[“000012″,”1016”],[“000013″,”959”],[“000014″,”901”],[“000015″,”880”],[“000016″,”800”],[“000017″,”665”],[“000018″,”693”],[“000019″,”646”],[“000020″,”610”],[“000021″,”543”],[“000022″,”443”],[“000023″,”289”],[“000024″,”219”],[“000025″,”133”],[“000026″,”100”],[“000027″,”84”],[“000028″,”81”],[“000029″,”149”],[“000030″,”231”],[“000031″,”524”],[“000032″,”744”],[“000033″,”859”],[“000034″,”894”],[“000035″,”848”],[“000036″,”861”],[“000037″,”892”],[“000038″,”860”],[“000039″,”832”],[“000040″,”842”],[“000041″,”809”],[“000042″,”744”],[“000043″,”650”],[“000044″,”711”],[“000045″,”583”],[“000046″,”483”],[“000047″,”305”],[“000048″,”220”],[“000049″,”115”],[“000050″,”98”],[“000051″,”56”],[“000052″,”127”],[“000053″,”269”],[“000054″,”481”],[“000055″,”787”],[“000056″,”1269”],[“000057″,”1467”],[“000058″,”1595”],[“000059″,”1613”],[“000060″,”1462”],[“000061″,”1446”],[“000062″,”1528”],[“000063″,”1545”],[“000064″,”1343”],[“000065″,”995”],[“000066″,”792”],[“000067″,”742”],[“000068″,”812”],[“000069″,”728”],[“000070″,”508”],[“000071″,”302”]]}

A single records contains the hour and the number of sessions:

[“000064″,”1343”]

What to the People of Albuquerque want from their Government?

By analyzing what the people of Albuquerque search for on the City website, we can come to some conclusions as to what they deem important or what concerns they have. To do so, we need the pages they looked at over a period of time. We can use the historic endpoint and pass pageTitle to find out.

https://shrouded-fjord-5635.herokuapp.com//historic?ids=ga:48754&dimensions=ga:nthHour&metrics=ga:pageviews&dimensions=ga:pageTitle&sessions&start-date=2015-02-21&end-date=2015-02-23

{“kind”:”analytics#gaData”,”id”:”https://www.googleapis.com/analytics/v3/data/ga?ids=ga:48754&dimensions=ga:pageTitle&metrics=ga:pageviews&start-date=2015-02-21&end-date=2015-02-23&#8243;,”query”:{“start-date”:”2015-02-21″,”end-date”:”2015-02-23″,”ids”:”ga:48754″,”dimensions”:”ga:pageTitle”,”metrics”:[“ga:pageviews”],”start-index”:1,”max-results”:1000},”itemsPerPage”:1000,”totalResults”:3090,”selfLink”:”https://www.googleapis.com/analytics/v3/data/ga?ids=ga:48754&dimensions=ga:pageTitle&metrics=ga:pageviews&start-date=2015-02-21&end-date=2015-02-23&#8243;,”nextLink”:”https://www.googleapis.com/analytics/v3/data/ga?ids=ga:48754&dimensions=ga:pageTitle&metrics=ga:pageviews&start-date=2015-02-21&end-date=2015-02-23&start-index=1001&max-results=1000&#8243;,”profileInfo”:{“profileId”:”48754″,”accountId”:”86004″,”webPropertyId”:”UA-86004-1″,”internalWebPropertyId”:”97739″,”profileName”:”2. http://www.cabq.gov All Traffic”,”tableId”:”ga:48754″},”containsSampledData”:false,”columnHeaders”:[{“name”:”ga:pageTitle”,”columnType”:”DIMENSION”,”dataType”:”STRING”},{“name”:”ga:pageviews”,”columnType”:”METRIC”,”dataType”:”INTEGER”}],”totalsForAllResults”:{“ga:pageviews”:”129345″},”rows”:[[“\”A Senior I Know\” Essay Contest — City of Albuquerque”,”30″],[“\”On the Commons\” Magazine Features Albuquerque Open Space — City of Albuquerque”,”3″],[“\”We Love Our BioPark\” Video Contest Launches — City of Albuquerque”,”3″],[“\”Website Transparency Team\”: Week of April 18, 2011 — City of Albuquerque”,”1″],[“‘Pacific Coral Reef’ Exhibit Now Open — City of Albuquerque”,”1″],[“(Memorial) Tribute Tree Program — City of Albuquerque”,”3″],[“(not set)”,”13″],[“01-28-2015 – 15-11-104F – Follow-up – On-Call Contractors – Department of Municipal Development — City of Albuquerque”,”1″],[“01-28-2015 – 15-12-107F – Follow-up – Health and Social Service Centers – Department of Family and Community Services — City of Albuquerque”,”1″],[“02-17-12 CASE# 12-202 Whistleblower Complaint against the Solid Waste Management Department — City of Albuquerque”,”1″],[“02-28-07 — 06-04-105F Follow-Up – Environmental Health Department Expenditures — City of Albuquerque”,”2″],[“04-16-09 — 07-204 – Final Investigative Report – Stolen Vehicles, SWMD — City of Albuquerque”,”1″],[“06-25-12 Case# 12-213 Collision Involving a Motor Coach Operator — City of Albuquerque”,”1″],[“06-25-2014 Case # 204 Water Authority — City of Albuquerque”,”1″],[“1 – Educated, literate residents — City of Albuquerque”,”1″],[“1-11-10 RYAB Facilitated Meeting Notes — City of Albuquerque”,”1″],[“1.1 Adult Educational Achievement Rates — City of Albuquerque”,”2″],[“10 – Basic needs provided for — City of Albuquerque”,”1″],[“10-Year Spending Trends — City of Albuquerque”,”8″],[“10.1 Residents Living in Poverty — City of Albuquerque”,”1″],[“10.2 Unemployment Rate — City of Albuquerque”,”1″],[“11 – The public is safe — City of Albuquerque”,”1″],[“11-19-07 — 07-202 – Investigation – Alleged Excessive Overtime Claim, ABQ Ride Department — City of Albuquerque”,”1″],

Conclusion

I have given you the endpoints. It is up to you to parse out the data. The code for that is on almost every post on this blog. Just grab any AJAX request up to JSON.parse() and grab results.rows[x].[0 through 1]. Something like that should be close.

 

Building Games with Leaflet, Turf and Albuquerque Open Data

24 Feb

I remember playing Risk as a kid. It was my first time playing a game that involved a map. When I got older, I loved playing games of world domination and strategy. As I build web maps, I often think, why couldn’t I make a game using Leaflet.js? In this post, I will describe how a game could be built with some sample code.

The Simple Game

There is no point to this game. It is an attempt to

  1. create pieces
  2. load data
  3. move the pieces with rules and
  4. accomplish a goal.

Winning the game is shown in the image below. You can test the game on JsBin.

I won the game by moving my piece in to a park.

I won the game by moving my piece in to a park.

The Game Components

The game board is made of a Leaflet.js map and Albuquerque Parks Data. The park data is loaded as a turf.polygon and pushed in to an array.

var url = “http://coagisweb.cabq.gov/arcgis/rest/services/public/fullviewer/MapServer/33/query?f=json&outSR=4326&outFields=*&where=1=1&#8221;;

http=new XMLHttpRequest();
http.open(“GET”, url, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {//Call a function when the state changes.
if(http.readyState == 4 && http.status == 200) {
console.log(“loading”);
var result= JSON.parse(http.responseText);
for(x=0;x<Object.keys(result.features).length;x++){
var pts=result.features[x].geometry.rings;
polygons.push(turf.polygon(pts));
}
L.geoJson(polygons).addTo(map);

}}
http.send();

Now that I have added the base layer, I want to create game pieces. It would be nice to also use Open Data for the players, but for now I will allow the user to click the map to drop a marker.

map.on(“click”,function(e){

var point=L.marker(e.latlng,{draggable:true}).addTo(map);

CODE BELOW GOES HERE
});

When the user clicks, a marker is added to the map and is set draggable. The game will allow the user to move the marker by dragging and attempt to land on a park. I want rules. The first rule I will create will only allow the user to drag the marker 30 pixels. This creates a problem. 30 pixels is a different distance based on the view level of the map. To fix this, I prevent the map from zooming so I know the 30 pixel fixed distance.

var map = L.map(‘map’, {center: [35.10418, -106.62987], zoom:15, touchZoom:false,scrollWheelZoom:false,doubleClickZoom:false,boxZoom:false
});

Now the user can pan but not zoom. If the user drags the marker too far, I need to set the marker to its original position. When the user starts dragging, we will get the lat,lng of it.

point.on(“dragstart”,function(z){
ll=this._latlng;
});

When the user finishes dragging the marker, we will check the distance. If it is greater than 30, we will alert them to the error and reset the marker position.

point.on(“dragend”,function(x){
if(x.distance>30){
alert(“You cannot move that far on a single turn”);
point.setLatLng(ll);}

Otherwise, we will allow the marker to move and test to see if the marker is inside of a park polygon. If it is, YAY, you win! else, keep going.

else{

for(k=0;k<polygons.length;k++){if(turf.inside(point.toGeoJSON(),polygons[k])){alert(“you made it to a park”);}}

}//else

});

This is the basic foundations of a game. To make it better, you would need to have turns and allow a computer controlled player to intervene. Maybe a zombie chase? Or you could network the game using websockets and allow users to use their real world position to play against each other(PDF) in a capture the flag or race to points type game.