Tag Archives: ESRI REST API

Go Library for ESRI REST

9 Jul

In my last post, I showed how to consume an ESRI REST geocoding service using Go. In this post, I will consume a feature service and query it. This post will take it one step further. We will put the code in a library so that anyone can use it in their applications. The full code is on GitHub.

Usually, the first line is package main, but we will change it to the name for our new package. then we can import the same libraries as in our geocoding sample.

package esrirest

import (
“fmt”
“net/http”
“io/ioutil”
“encoding/json”

)

The structs that we will create will be different as well. In the geoocoding example we wanted candidates. When running a query, we want features.

type query struct{
Features []features
}

ESRI GeoJSON uses Attributes instead of properties, so we will create a map of type interface for the attributes in the feature. This allows us to grab any data types without having to hardcode them as we did in the geocoding example (address, location {x,y}).

type features struct{
Attributes map[string]interface{}
}

In case we need it later – I ever get around to adding more functionality – I will put the variable data outside the function.

var data query

Now, instead of writing a main() function, we will write a function Query – an not include a main in this program because it is a library, and not meant to be executed stand alone. The Query functino will take a string (URL with parameters) that points to the ESRI service. It returns a slice of features. The rest of the code looks exactly like the geocoding example.

func Query(s string) []features {

response, err := http.Get(s)
if err != nil {
fmt.Printf(“%s”, err)

} else {
defer response.Body.Close()
c, _ := ioutil.ReadAll(response.Body)
json.Unmarshal(c,&data)

}
return data.Features
}

The one difference is that the function returns the slice of features – data.Features. You now have a functioning library that you can use in any program.

Use the Library

I will now show you how to use the library.

Run the command:

go get github.com/PaulCrickard/Go/esrirest

This will grab the plugin and install it locally for you. Now write your application.

Declare the package to be main and import the needed libraries. In this case we only need fmt and our new library that we created, downloaded and installed.

package main

import (
“fmt”
“github.com/PaulCrickard/Go/esrirest”

)

If you look in your pkg directory, you will see a folder with your OS type (windows_amd64) then  github.com, PaulCrickard, Go and a file esrirest.a. That is the path you use for the import.

Create your main function and now just call the function in our library passing the URL.

func main(){
//Pass query parameters in URL. MUST HAVE —–> f=json
d :=esrirest.Query(“http://coagisweb.cabq.gov/arcgis/rest/services/public/PublicArt/MapServer/0/query?where=1=1&outFields=*&f=json”)
}

All of the data is now in d. We can now access features and attributes.

fmt.Println(d[0].Attributes[“TITLE”]

Or, we can iterate through all the features and print the titles.

for i:=0; i<len(d); i++{
fmt.Println(d[i].Attributes[“TITLE”])
}

What if you do not know what the attribute names (keys) are? You can iterate through and grab them.

for key, value := range d[0].Attributes{
fmt.Println(key,value)

}

This will print out the key and value for the first feature. Throw this inside your for loop above to go through each feature.

You can grab just the keys by using:

for key, _ := range d[0].Attributes{
fmt.Println(key)

}

Or print the values:

for key, _ := range d[0].Attributes{
fmt.Println(d[0].Attributes[key])

}

Now you know how to crate a library, how to install it and how to use it in another application.

 

Aerial Imagery of Albuquerque and Surrounding Counties

10 Mar

When you need an aerial image, do you go to Google and take a screenshot? If you live in Albuquerque or the surrounding area, you no longer need to do that. The City of Albuquerque publishes aerial imagery of Albuquerque and the surrounding areas as open data. In this post, I will show you how to retrieve an image.

The Service

The Imagery Service provides a list of all the available years.

The historic data is a great addition.

The historic data is a great addition.

For this tutorial, click on Imagery/Aerials2014. At the bottom of the page, choose Export Image. You will see the full extent of the imagery and a form below.

Capture2

To get an image, you need to enter the lower left and upper right corner coordinates of a bounding box in NAD83(HARN) / New Mexico Central (ftUS). I have no clue what this would be. You can change it to WGS84 (4326) as we have done in previous examples, but still, how would I know the coordinates without clicking on a map and checking the points? The form is pretty much useless to me. Our application will provide a better GUI – map based – to the form.

The Application

Our application will be a Leaflet.js map that allows the user to zoom to their desired location. The map window will be used to select what is exported from the imagery service.

The Logic

The application is a basic Leaflet.js map. We will add an onClick function that grabs the image.

map.on(“click”,function(){

getImage();

});

The getImage() function will make an AJAX call to the service. It will pass the coordinates of the map window as parameters to get back an image in a popup. You can then right click and save image as…

function getImage(){

b=map.getBounds();
var bounds=b._southWest.lng+”,”+b._southWest.lat+”,”+b._northEast.lng+”,”+b._northEast.lat;

var url = “http://coagisweb.cabq.gov/arcgis/rest/services/Imagery/Aerials2014/ImageServer/exportImage&#8221;;
var params=”f=json&bbox=”+bounds+“&bboxSR=4326&size=1028,1028&imageSR=4326”;

http=new XMLHttpRequest();
http.open(“POST”, url, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {//Call a function when the state changes.
if(http.readyState == 4 && http.status == 200) {

result= JSON.parse(http.responseText);
window.open(result.href);

}}

http.send(params);

}

The above code starts by assigning the map bounds to a variable. We then create a bounds string in the format the service wants – lower left corner to top right corner. We pass the bounds string in to the parameters string and make the request.

The result contains an href property that is the URL to the image. We open a popup to the URL so the user can save the image.

Map on the left. Click. Image popup on the right.

Map on the left. Click. Image popup on the right.

Now the user can zoom to the desired extent and click the map to get back an image. Much easier than finding the coordinates and entering them in to the form.

 

Reverse Geocoding a Line

3 Mar

In Geocoding with ABQ Open Data, I showed you how to geocode an address and how to reverse geocode a point when the user clicks on the map. Geocoding is an operation performed on points or addresses, so what do we do with lines? In this post I will show you how I am geocoding a line.

The Need

I have an application that allows a user to select a location for where they would like to perform some construction work and need a permit. The application started by allowing the user to specify a point or address. As I met with the users of the application, it turned out that having line features was crucial. Work may be performed along a stretch of road and the approvers of the permit wanted to be able to find at least the starting address when looking at the data in a tabular format – mostly for things like how many permits are on Central Ave.

The Solution

Alert box showing the start and end addresses of a line

Alert box showing the start and end addresses of a line

The solution uses the same reverse geocoding process as my previous post, but parses a line feature in to points. I am using the leaflet.draw plugin to allow the user to draw the line on a map. The draw plugin has a draw:created event that searches to see the type of object drawn. My code runs in the if (type == ‘polyline’) block.  The code below converts the layer drawn to GeoJSON and then the GeoJSON coordinates to a string split on the comma. Using the string at index 0 and 1, I have the starting point of the line. Using the index of length-1 and length-2 I can grab the last point. I use length because the user can draw a line segment with an unlimited number of points so I will not know where the line ends and do not want to assume a two point line. when I have the points, I call reverseGeocode().

var tojson=layer.toGeoJSON();
var x=String(tojson.geometry.coordinates).split(“,”);

//start point
var lt=parseFloat(x[1]);
var ln=parseFloat(x[0]);
reverseGeocodeLineFrom(ln,lt);

//endpoint
var ltend=parseFloat(x[x.length-1]);
var lnend=parseFloat(x[x.length-2]);
reverseGeocodeLineTo(lnend,ltend);

The reverseGeocode function just makes an AJAX call to the ESRI REST API service for my geocoder passing the point and parsing the response.

function reverseGeocodeLineFrom(a,b){

var urlgeocode=”http://servername/ArcGIS/rest/services/wgs84locator/GeocodeServer/reverseGeocode&#8221;;

var geocodeparams = “location=”+a+”,”+b+”&distance=1000&f=json”;
var http;
http=new XMLHttpRequest();
http.open(“POST”, urlgeocode, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {//Call a function when the state changes.
if(http.readyState == 4 && http.status == 200) {

addressAsJSON=JSON.parse(http.responseText);
alert(addressAsJSON.address.Street);
}
}
http.send(geocodeparams);
}

Now I have a street address for the start and end of my line.

ESRI REST and Python: pandas.DataFrame

13 Jan

I have been working in JavaScript a lot recently and have missed coding in python. I wanted to work on something that would have real world value so I decided to see if I could load an ESRI REST endpoint in to a pandas DataFrame. This post will only scratch the surface of what you can do with data in python but should give you an idea of what is possible and allow you to imagine some really interesting possibilities. I have posted all of the code as a static IPython notebook and on Python Fiddle for easy copy+paste.

Getting Data

We are going to start by grabbing some open data from the City of Albuquerque – I will use the Crime Incidents REST service. The first step is to make the request and read it.

import urllib, urllib2
param = {‘where’:’1=1′,’outFields’:’*’,’f’:’json’}
url = ‘http://coagisweb.cabq.gov/…/APD_Incidents/MapServer/0/query ?’ + urllib.urlencode(param)
rawreply = urllib2.urlopen(url).read()

If you print rawreply, you will see a long string. We know it is a string because if you print rawreply[0] you will see { as the result. To confirm it, you can type(rawreply) and get back str.

Working with the Data

Now we need to convert it to JSON so that we can do something with the data.

import simplejson
reply = simplejson.loads(rawreply)
print reply[“features”][0][“attributes”][“date”]
print reply[“features”][0][“attributes”][“CVINC_TYPE”]

The above code will return something like

1405468800000
LARCENY ALL OTHER

How many features do we have? 24,440.

print len(reply[“features”])
reply[“features”][:3]

The above code will give you the count and show you the first three results. You can grab subsets of the data by splicing. If you want records 2 through 4, you can reply[“features”][2:5].

Let’s get to the good stuff.

Pandas DataFrame

JSON is great, but let’s put the data in to a table (DataFrame). First, a disclaimer. ESRI REST services return a deeply nested JSON object. Converting this to a DataFrame is more difficult than:

import pandas.io.json as j
jsondata = j.read_json(the URL)

But it is not much more difficult. We just need to recreate the JSON object grabbing what we are interested in – the attributes of the features. We need a loop that will  create an array of dictionaries. Each dictionary will be the attributes of a single feature.

count = 0
myformateddata=[]
while (count < len(reply[“features”])):
mydict={}
for key, value in reply[“features”][count][“attributes”].iteritems():
mydict[key]= value
myformateddata.append(mydict)
count = count + 1

The code above initializes a counter, array and a dictionary. It then loops until there are no more features. The loop reads the attributes of the feature and creates a dictionary for each attribute and its value. Now we have an array of dictionaries. If you len(myFrame) you will get 24,440. That is the same number of records we had in our JSON object.

We have all our data and it is an array. We can now create a DataFrame from it.

from pandas import DataFrame as df
myFrame = df(data=myformateddata)
myFrame

The code above imports the tools we need, assigns the frame to a variable and then prints it. You should see a table like the image below.

DataFrame with Crime Incidents from ESRI REST endpoint

DataFrame with Crime Incidents from ESRI REST endpoint

Data Manipulation

So what can we do with a DataFrame? First, let’s save it. We can export the DataFrame to Excel (The City of Albuquerque provides open data in several formats, but other sources may only provide you with the REST API so this would be a valuable method for converting the data).

from pandas import ExcelWriter
writer = ExcelWriter(‘CrimeIncidents.xlsx’)
myFrame.to_excel(writer,’Sheet1′,index=False)
writer.save()

Great! the data has been exported. Where did it go? The code below will return the directory with the file.

import os
os.getcwd()

You can open the Excel file and work with the data or send it to someone else. But let’s continue with some basic manipulation in the DataFrame.

How many of each type of incident do we have in our data?

myFrame[“CVINC_TYPE”].value_counts()

Incidents Summarized

Incidents Summarized

Now I would like a barchart of the top 5 most frequent Incident Types.

import matplotlib.pyplot as plt
incidentType=myFrame[“CVINC_TYPE”].value_counts()[:5]
incidentType.plot(kind=’barh’,rot=0)

My Bar Chart of the 5 most common Incidents

My Bar Chart of the 5 most common Incidents

Lastly, let’s filter our results by date.

import datetime
import time
myDate=datetime.datetime(2015,1,10)
milli=time.mktime(myDate.timetuple()) * 1000+ myDate.microsecond / 1000

The above code creates a date of January 10, 2015. It then converts it to milliseconds – 1420873200000. If we pass the date to the DataFrame we can grab all the Incidents after January 10, 2015.

myFrame[(myFrame.date>milli)]

DataFrame for Incidents After January 10, 2015

DataFrame for Incidents After January 10, 2015

Now you know how to connect to an ESRI REST endpoint, grab the data, convert it to JSON and put it in a DataFrame. Once you have it in the DataFrame, you can now display a table, export the data to Excel, plot a barchart and filter the data on any field.

Open Data and Projections

13 Jan

I am not a geographer, nor am I a math wiz. So for me, projections have been a constant source of pain and suffering. I have opened my mouth and said that WGS84 should be the default for all GIS data. It didn’t go over well – mostly because I think the old school GIS folks have just always done things a certain way and will not stop. Apparently, I am not the only person that hates projections. Calvin Metcalf hates projections too. In the real world, we are the minority and are stuck dealing with them. So in this article I will show you how the ESRI REST API makes it much simpler.

The Open Data

I will start by connection to some Albuquerque Open Data. In this example, I will query the Parks REST service. First, open the link for the parks service and notice that the spatial reference is 102100  (3857). This is WGS 1984 Web Mercator. If you query where 1=1, you can see coordinates like [-1.1855471312199999E7, 4173947.5449]. I want to use WGS84 (4326) so I need coordinates like [35.10418, -106.62987]. We could use a library like Proj4js to convert, but the API already has the functionality we need built right in. On the query form, add 4326 to the output spatial reference field and now you (-1.xx , 417.xxx) becomes [-106.xxx, 35.xxx]. This is what we need.

When you query, you may also want to pass in coordinates from a different spatial reference. You can do this by changing the input spatial reference to 4326. The server will handle the conversion and return the correct feature for you.

An Example

I will start by creating a simple Leaflet.js map that returns coordinates in 4326 on a map click.

<html>
<head><title>Projections</title>
<link rel=”stylesheet” href=”http://cdn.leafletjs.com/leaflet-0.7.2/leaflet.css&#8221; />
<style>
html, body, #map {
padding: 0;
margin: 0;
height: 100%;}

</style>
</head>
<body>
<script src=”http://cdn.leafletjs.com/leaflet-0.7.2/leaflet.js”></script&gt;
<div id=”map”></div>
<script>
var map = L.map(‘map’,
{center: [35.10418, -106.62987],
zoom:15 });
L.tileLayer(‘http://{s}.tile.osm.org/{z}/{x}/{y}.png’).addTo(map);
map.on(“click”,function(e){
alert(e.latlng);
});
</script>
</body>
</html>

To make a query, we need to replace the alert() function. I will use an AJAX query to the Parks REST endpoint.

First, we need to capture the (lat,lng) in a format we can use. You could split the e.latlng string but that is too much trouble. I will create a marker when the user clicks. I will not add the marker to the map, but rather, I will convert it to GeoJSON. Then I will create a parameter string from several other strings.

var marker=L.marker(e.latlng);
coords=marker.toGeoJSON();
q1=”inSR=4326&outSR=4326&f=json&geometryType=esriGeometryPoint&geometry=”;
q2=String(coords.geometry.coordinates[0]);
q3=”,”;
q4=String(coords.geometry.coordinates[1]);
var params=q1.concat(q2,q3,q4);

In the above code, I grabbed the click e.latlng and passed it to create a marker. The coords variable holds a GeoJSON representation of the marker.

The variable q1 holds my parameters:

  1. inSR = 4326 – this is the projection I want to use
  2. outSR = 4326 – I want the server to give me back the same, not 3857
  3. f=json – I want json, not the HTML
  4. geometryType = esriGeometryPoint – I am taking a point and want the polygon (park) it is in

The last parameter is the geometry.  I specified a point so the server expects two coordinates. I use coords.geometry.coordinates[0] and coords.geometry.coordinates[1] – These are the values for longitude and latitude from the GeoJSON representation of the marker.

Now, I will make the AJAX call and close the map.on function.

var params=q1.concat(q2,q3,q4);
var url = “http://coagisweb.cabq.gov/arcgis/rest/services/public/recreation/MapServer/0/query?&#8221;;
http=new XMLHttpRequest();
http.open(“POST”, url, true);
http.setRequestHeader(“Content-type”, “application/x-www-form-urlencoded”);
http.onreadystatechange = function() {
if(http.readyState == 4 && http.status == 200) {
theGeom= JSON.parse(http.responseText);
console.log(theGeom.features[0].attributes.PARKNAME);
console.log(JSON.stringify(theGeom.features[0].geometry.rings));
}
}
http.send(params);
}); // close map.on

Now you should have a map that waits for a click, then queries the REST endpoint passing a point in 4326 and gets back a park name with coordinates in 4326.

I zoomed in to Robinson Park.

I zoomed in to Robinson Park.

Clicking Robinson Park gave me the screenshot below.

Robinson Park with Rings

Robinson Park with Rings

You do not have to be an expert on projections, just use the tools provided and pass what you want to input and what you want sent back to you as parameters in your queries.

Add Feature With Different Projection

10 Dec

In my previous post, I showed how to query from one coordinate system and get the results in another. This was all in an effort to add a feature to an ESRI feature class in 2903 using leaflet.js in 4326. I remembered that some services allow you to specify an inSR, however, that option was missing in the addFeature capabilities.

I took a minute to read the documentation and viola! you can specify an input coordinate system. So now you can go from 4326 using leaflet.js to any coordinate system in ESRI using REST. All you need to do is pass a spatial reference in your geometry and ESRI will handle the conversion for you.

I have 4326 (lat,long) and want to send to 2903:

[{“geometry”:{“x”:-106,”y”:35,”spatialReference” : {“wkid” : 4326}}}]

This tells the service that I  know you are in WKID:2903 but I am giving you WKID:4326, deal with it. And it does!

I will admit that I am awful with projections and coordinate systems and this is a great way for me to avoid them…..for now.

ESRI REST API: GeoCoding OR OpenStreet Map

10 Jun

UPDATE: OpenStreet Map code at bottom.

 

Saw a Tweet saying that the ArcGIS REST API for Geocoding is moving to a new URL. I use Google when I just needed a single Lat,Long and ArcPy to Geocode CSV’s, but I decided to check out the REST API and throw together a Python script from my old Google one – which I got from Foundations of Python Network Programming.

Not much to the script – pass parameters to a URL and grab the JSON. You can go to the HELP for more info about the parameters – like setting outfields=*;

Here is the script (it grabs the first result):

import urllib, urllib2, simplejson
import csv

param = {‘Address’: ‘400 Roma SE’,’City’:’albuquerque’,’Region’:’nm’,’Postal’:’87102′,’outFields’:’location’,’f’:’pjson’}
url = ‘http://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/findAddressCandidates?&#8217; + urllib.urlencode(param)
rawreply = urllib2.urlopen(url).read()
reply = simplejson.loads(rawreply)

print “Long: ”
print reply[“candidates”][0][“location”][“x”]

print “Lat: ”
print reply[“candidates”][0][“location”][“y”]

 

OpenStreet Map is almost the same, just change the URL and the parameters.

import urllib, urllib2, simplejson
import csv

param = {‘q’: ‘400 roma, albuquerque’,’format’:’json’,’addressdetails’:’1′}
url = ‘http://nominatim.openstreetmap.org/search?&#8217; + urllib.urlencode(param)
rawreply = urllib2.urlopen(url).read()
reply = simplejson.loads(rawreply)

print reply[0][“lat”]
print reply[0][“lon”]