Archive | January, 2013

JSON to Shapefile

31 Jan

Toronto has a bunch of open data – which is awesome! Someone I follow on Twitter was looking to convert some parking JSON to a Shapefile. I haven’t used it much, but python has a JSON module in 2.7 and SimpleJSON in 2.5. And of course, writing shapefiles is simple with shapefile.py.

JSON is a dictionary. Once it is loaded in a data variable, you can call items by the key. So for the carparks, I could call: data[“carparks”][i][“id”] or data[“carparks”][i][“lat”], where i is the index or the array. Then iterate, while i < len(data[“carparks”]).

Here is the full code:

import shapefile
import json

json_data=open(‘greenPParking.json’)
data = json.load(json_data)

w=shapefile.Writer(shapefile.POINT)
w.field(“id”)
w.field(“address”)
w.field(“lat”)
w.field(“lng”)
w.field(“rate”)
w.field(“rate_half_hour”)
w.field(“carpark_type”)
w.field(“carpark_type_str”)
w.field(“capacity”)
w.field(“max_height”)
w.field(“payment_options”)

i=0 # should be changed to — while (i < len(data[“carparks”]))
while (i<243):

w.point(float(data[“carparks”][i][“lng”]),float(data[“carparks”][i][“lat”]))
w.record(data[“carparks”][i][“id”],data[“carparks”][i][“address”],data[“carparks”][i][“lat”],data[“carparks”][i][“lng”],data[“carparks”][i][“rate”],data[“carparks”][i][“rate_half_hour”],data[“carparks”][i][“carpark_type”],data[“carparks”][i][“carpark_type_str”],data[“carparks”][i][“max_height”],data[“carparks”][i][“capacity”],data[“carparks”][i][“payment_options”],data[“carparks”][i][“rate_details”])
i+=1

prj = open(“csvSHP.prj”, “w”)
epsg = ‘GEOGCS[“WGS 84”,’
epsg += ‘DATUM[“WGS_1984”,’
epsg += ‘SPHEROID[“WGS 84”,6378137,298.257223563]]’
epsg += ‘,PRIMEM[“Greenwich”,0],’
epsg += ‘UNIT[“degree”,0.0174532925199433]]’
prj.write(epsg)
prj.close()

w.save(“greenPParkingSHP”)
json_data.close()

Big Data: My First Kaggle. My First SciKit Learn.

25 Jan

titanic

Recently, I saw a tweet that linked to the Kaggle blog. I had no idea what Kaggle was. Apparently, it’s making data science a sport. The reason the tweet caught my attention was because I had started reading “Data Mining: Practical Machine Learning Tools and Techniques.” Googling “python machine learning” brought me to scikit learn. I remembered the PyData conference posted several videos on machine learning and had three videos on Scikit-learn. Looked like fun, so I entered the Titanic Kaggle Competition.

The competition provides a set of training data (people on the Titanic and whether they lived or died) and a set of test data. I still don’t know too much about the techniques of machine learning, but I was able to figure out how to run LogisticRegression and LinearSVC. Slapping together two simple python programs,  I scored 75% with SVC and 76% with Regression. I did have to fudge the training data to scrub text strings and replace missing data with 0’s. This probably affected my model – also used defaults – but I was happy to even get it to run and format the data.

Now I am motivated to learn more about the methods and parameters of the models. A side note – the Regression model gave probabilities for whether a person lived or dies, I accepted the model if it was greater than .50 probability.

Here is my code for the SVC. The Regression is almost identical.

import csv as csv
import numpy as np
from sklearn.svm import LinearSVC

train=csv.reader(open(‘titanictrain.csv’,’rb’)) #open file
headerx=train.next()

x=[]
for row in train:
    x.append(row)
x=np.array(x)
category=csv.reader(open(‘titanictarget.csv’,’rb’)) #open file
headery=category.next()

y=[]
for row in category:
    y.append(row)
y=np.array(y)

clf=LinearSVC()
clf = clf.fit(x,y)

test=csv.reader(open(‘test.csv’,’rb’))
headert=test.next()
new=[]
for row in test:
    new.append(row)
new=np.array(new).astype(np.float)

resultset=clf.predict(new).astype(np.float)
np.savetxt(‘results.csv’,resultset,delimiter=”,”)

 

Launch External Script in ArcGIS Desktop Explorer

24 Jan

arcgis_explorer_15001

I have started to toy with the ArcGIS Desktop Explorer API. My first few attempts at functionality required only the use of C# Libraries and not the actual SDK Library – with the exception of getting the button in.

If you have the SDK installed, you can start a new project in Visual Studio Express and select the GIS Button. This first example will allow you to execute a script or program on your machine. Where it tells you to insert your code, enter the following line:

 System.Diagnostics.Process.Start(“myPythonFile.py”);

This will run the python file. You could also use launch system applications like calc or notepad.

My next test was to launch a website from a button.  Same as the above, except enter the URL of the site. This will open the page in your default browser.

System.Diagnostics.Process.Start(“http://google.com&#8221;);

One thing I had trouble with was how do I add several buttons in a single Namespace? I copied the button that was created by default and pasted it in to the namespace. Then you need to change the name and the functionality. Lastly, you need to edit the AddIns.xml file. Copy the first button and paste it. Then modify the values to math your new button. I have put the values that MUST change in bold. You should change the captions too. Also, drag any images to the images folder. 32×32 for large and 16×16 for small.

<button
id=”Button1
assembly=”WebLaunchButtons.dll”
className=”WebLaunchButtons.Button1
caption=”Google”
tooltip=”Quick Access to Google.”
helpString=”http://mydomain.com&#8221;
largeImage=”Images\google.png”
smallImage=”Images\google_SmallImage.png” />

<button
id=”Button2
assembly=”WebLaunchButtons.dll”
className=”WebLaunchButtons.Button2
caption=”Yahoo!”
tooltip=”Quick Access to Yahoo.”
helpString=”http://mydomain.com&#8221;
largeImage=”Images\yahoo.png”
smallImage=”Images\yahoo_SmallImage.png” />

Now you will have 2 buttons in a frame with a single group title – you set this in the initial project settings.

My last example will connect to an FTP site, grab a shapefile, and load it. The code is a mashup of code from MSDN and ESRI as well as some modifications on my part.

public override void OnClick()
        {
            m_disp = ESRI.ArcGISExplorer.Application.Application.ActiveMapDisplay;

            FtpWebRequest request = (FtpWebRequest)WebRequest.Create(“ftp://ftp1.freebsd.org/pub/FreeBSD/README.TXT”); //this would be to a shapefile but I dont have an FTP server.
            request.Method = WebRequestMethods.Ftp.DownloadFile;

           
            //request.Credentials = new NetworkCredential(“me”, “mypassword”);

            FtpWebResponse response = (FtpWebResponse)request.GetResponse();

            Stream responseStream = response.GetResponseStream();
            StreamReader reader = new StreamReader(responseStream);
            System.IO.StreamWriter file = new System.IO.StreamWriter(“c:\\downloaded.txt”); //your shapefile
            file.WriteLine(reader.ReadToEnd());
            
            reader.Close();
            response.Close();
            file.Close();

            FeatureLayer fl = FeatureLayer.OpenShapefile(“C:\\downloaded.shp”); // the file you saved
            fl.Renderer.SetSymbol(CreateSymbol(fl.GeometryType));
            m_disp.Map.ChildItems.Add(fl);
            m_disp.ZoomTo(fl.Extent);

            
        }

        private Symbol CreateSymbol(GeometryType geomType)
        {
            //Create a new symbol for the feature layer data being added based on feature type
            Symbol sym = null;
            switch (geomType)
            {
                case GeometryType.Point:
                    sym = Symbol.Marker.Pushpin.LightBlue;
                    break;
                case GeometryType.Polygon:
                    sym = Symbol.CreateFill(System.Drawing.Color.Aqua, System.Drawing.Color.Black);
                    break;
                case GeometryType.Polyline:
                    sym = Symbol.Line.Solid.Purple;
                    break;
            }

            return sym;
        }

    }

In my next post, I will show how to send an SMS text message from GIS.

Realtime Bus Map Using OpenLayers and KML

12 Jan

20130112-093919.jpg
The City of Albuquerque posts a KML file every minute that shows the location of a bus. Using OpenLayers and Open Street Map, I built a map to display route 157.

I needed the KML to be stored locally do I created a PHP script to grab the file. In the HTML, I called the script using . It runs, saves the KML and then OpenLayers displays it.

I would like to use $_GET to allow the passing of a route # to find your bus and a QRCode on the bus sign.

See the map.