The open-sourcing of Mobile Oxford

The open-sourcing of Mobile Oxford has begun in earnest. On our list of things to do we have choosing a license, producing documentation, and splitting out the Oxford-specific parts.

To differentiate between the Oxford instance and the code it’s running on we’ve decided to call the project Molly.

Licensing

We’re as yet unsure as to which license we’ll be releasing Molly under, though we’re currently favouring a permissive license over a copyleft license.

To help us decide we’ll be meeting with OSS Watch (the higher education open-source software advisory service) tomorrow to help us make an informed decision. OSS Watch also provide background on a number of open-source licenses, which has proved useful in getting us not-so-legally-aware types up to speed.

Choosing the right license is essential to ensure that we foster as much input as possible from other parties. By way of example, Molly is designed to integrate rather heavily with an institution’s existing systems, and it’s possible that said institution might not want to publish how those interfaces work. As I understand it (and remember, I am not a lawyer) such an interface would constitute a derivitive work and require publishing were we to choose the AGPL. Additionally, the GPL is effectively permissive if used for a networked service as the software itself is never distributed – yet the name may still hold some ‘scare factor’ and thus put off contributions from some institutions.

Sakai is licensed under the Educational Community License, and as Molly is intended to provide strong Sakai integration we should consider how we position ourselves alongside.

Splitting out the Oxford-specific functionality

Mobile Oxford was initially intended to be a ‘demonstration location-aware application,’ but has serendipitously turned into something more. Being a demonstration, there hasn’t always been a strict separation between functionality and data sources. We intend that Molly will provide the user interface and data model for various applications, with the implementer creating a ‘provider’ to hook it up to a local system.

So far I’ve pulled out the contact search, creating three providers. These are our original screen-scraping implementation, another using an IP-restricted web service, and an LDAP-based solution for MIT’s people directory (based on their open-source MIT Mobile Web). The core functionality handles pagination, linking to further details and display; the provider can pass in a list of results for a given query and retrieve specific people based on some unique identifier. The interface between them is based on the LDAP attributes defined in RFC 4519 and comprises three methods and two attributes, making implementation relatively easy.

Contact search is one of the simpler parts of the site, so it’s going to take a fair bit of effort to provide a similar level of abstraction for the remainder of the site’s functions.

Documentation

We’ll be using Sphinx for Molly’s documentation. Sphinx is the closest thing to a standard in Python documentation, being used by many Python-based projects including Python itself and the Django project. Sphinx is specifically intended for documenting Python projects, including support for cross-documentation links and coverage reporting.

Documenting is also proving very useful in formalising the internal interfaces and exposing previous poor design decisions. There’s been at least a couple of occasions so far where I’ve documented how it should have been done and then had to refactor the code to bring it in line.

Live bus locations from ACIS/OxonTime

I spent a day a little while back investigating how to get the raw data behind OxonTime’s live bus locations. Here’s how it works.

Please be aware that what follows is the result of a purely academic investigation, and that before using this information it may be worth contacting Oxfordshire County Council to discuss your plans, particularly if you’re going to distribute the results of your endeavours.

Performing a request

By monitoring the HTTP requests made by the applet we can deduce that it fetches data from a particular resource with parameters provided in the querystring. The resource is located at http://oxfordshire.acislive.com/pda/mainfeed.asp, and takes the following parameters:

type

One of STOPS, INIT or STATUS. The first retrieves a list of stops and their locations for a given area. The latter two both return a list of buses, with the latter being used for updates.

maplevel

I believe this only accepts the values 0 through 3, with nothing returned for 0 and 1. The results for 2 and 3 seem identical, so there seems little point in varying it.

SessionID

This seems to be a misnomer as it specifies the area you want to enquire about. We’ll explain how to convert to and from these numbers later.

systemid

Always 35 as it gets unhappy if you change it.

stopSelected

This expects an ATCO code yet seems to be ignored, so you may as well leave it as 34000000701.

vehicles

A comma-separated list of vehicle identifiers you currently believe to be in the area you’re enquiring about. This is only passed when type=STATUS, and lets you find out when the given buses have left the area.

Format of responses

The response in each case is a pipe-delimited list of values, with the first being the action the client should perform in updating its state. The things you may expect are:

STOP

Retrieves a list of stop locations in the area. Nearby stops are collected into one line.

NEW

This signifies that a bus is to be found at the given location.

DEL

These only appear when type=STATUS and signify that a bus is no longer at the location. If the bus is still within the requested area there will be a subsequent corresponding NEW action to give its new location. If it has left there will be no such NEW action. The buses that appear here will be a subset of those provided in the vehicles parameter of the querystring.

Now we know the form of the response, let’s see the values returned for each action. We’ll start with STOP.

STOP actions

A STOP action has the form:

STOP|35|naptan-codes|x,y|stop-names|stop-bearings|count

Here’s an example:

STOP|35|693123456^693234567^693345678|12,413|Banbury Road B1^Banbury Road B2^Banbury Road B3|172^179^340|3

As mentioned earlier, stops that are close to one another (e.g. on opposite sides of the road, or a ‘lettered’ group) are collected together, with count giving the number of stops at this location.

naptan-codes, stop-names and stop-bearings are each caret-delimited lists with fairly obvious contents. stop-bearings are given in clockwise degrees from grid north.

x and y give pixel offsets from the top-left corner of the displayed area. More on this later.

I have no idea what the 35 signifies, and currently assume it’s something to be with systemid.

A note on bus stop identifiers

The NaPTAN (National Public Transport Access Nodes) database provides two classes of identifiers, ATCO codes and NaPTAN codes. ATCO codes are upto 12 characters in length, whereas NaPTAN codes consist of nine digits. OxonTime predominantly exposes the latter; these are the numbers beginning ‘693’ displayed at bus stops. However, ATCO codes are used for the currentStop parameter and are accepted elsewhere in place of their equivalent NaPTAN codes.

The NaPTAN database is currently maintained under license from the Department for Transport by Thales. Access requires a license, which may come with a fee for commercial use. However, you may be interested to note that NaPTAN data is finding its way into OpenStreetMap.

NEW actions

These have the form:

NEW|identifier|orientation|service-name|Operators/common/bus/1|y,x

Here’s an example:

NEW|1024|4|X5|Operators/common/bus/1|45,302

identifier serves to keep track of buses between requests. I don’t know whether it has some further meaning outside of this API. orientation is an integer between 1 and 8 inclusive, being N, NE, E, SE, S, SW, W, NW respectively. service-name is the same as is used on the rest of the site (e.g. ‘S1’, ‘5’, ‘TUBE’). The next bit seems constant and can probably be safely ignored. Finally the offsets are given, only this time with y first; I have no idea why.

DEL actions

These have the form:

DEL|identifier|

Here’s an example:

DEL|1024|

These identifiers match up with those given in NEW actions. Note the trailing pipe.

By periodically making requests with type=STATUS one can process the returned lines of a stream of commands describing how to update the local state. This makes client implementation easier as you are effectively applying a diff, as opposed to having to compare new to old.

The co-ordinate system

First off, I’d better give you a disclaimer. This API and its associated co-ordinate system is very specific to the applet that is its only intended client. As such, the co-ordinate system provides exactly what it needs and no more.

Locations are addressable using a combination of SessionID — hereafter known as map number for clarity — and an x and y pixel offset from the top-left corner of that tile.
The maps are each 418 pixels square, and are arranged in a grid aligned with the British National Grid. The important thing to note about this is that grid North is not the same as true North, and that if you intend to plot these things on (for example) a Google or OpenLayers map, you’ll need to get your projections right.

The maps seem to be numbered somewhat arbitrarily, as shown by this map of bus stops and their associated map numbers. Colours are based on a hash of the number of the map they appear on.

A map showing the relative locations of bus stops and the ACIS map numbers for each map.

These were found by requesting bus stops for all map numbers between 2500 and 3000, so are likely not complete. Predicting the map numbers for areas beyond these seems non-trivial or prone to error.

Doing the conversion

Edit: The following are for zoom-level 3 maps. Lower map numbers are at zoom-level 2 and cover nine times the area, so it probably makes more sense to retrieve and parse those.

To help you on your way, here is a Python dictionary which maps between map numbers and their top-left corners expressed as metres East and North of the SV square on the British National Grid. A pixel is equivalent to about 2.2×2.2 metres, as given by scale, making each map about 920 metres square.

map_numbers = {
    2507: (445998.30384769646, 204584.56186062991),
    2511: (446915.61022902746, 204584.49926297821),
    2512: (446913.71674095385, 205502.67433143588),
    2513: (446914.66775580525, 206418.97178315694),
    2516: (447831.1147245816, 205501.4561684193),
    2517: (447831.2934340348, 206419.14371744529),
    2520: (448749.2218840057, 205501.31610451243),
    2521: (448748.08603126393, 206418.72288576054),
    2522: (448749.21670514799, 207337.62523250855),
    2537: (448748.23059583915, 210084.58299463647),
    2538: (448747.09615575505, 211001.00119758685),
    2543: (450582.89109881676, 200918.25920371327),
    2545: (450582.58945117606, 202753.0387530663),
    2546: (450582.18354506971, 203668.85981883231),
    2548: (451497.38359153748, 201836.31023917606),
    2549: (451498.11202925985, 202752.26514351295),
    2550: (451499.38388189324, 203669.3077042877),
    2551: (452417.47236742743, 200918.93886234396),
    2552: (452416.98231942754, 201835.54957236422),
    2554: (452414.24108836311, 203668.59070562778),
    2557: (449664.20383959071, 206418.02172485131),
    2560: (450580.70403987489, 205501.67253490919),
    2561: (450580.96866136562, 206419.57452165778),
    2562: (450592.76460073108, 207336.75147627734),
    2563: (451499.11722530029, 204585.22058827255),
    2564: (451500.02145752736, 205501.68224598444),
    2565: (451499.00067467656, 206418.26238617001),
    2567: (452415.21798651741, 204584.90006837764),
    2568: (452415.5239759339, 205501.35499095405),
    2569: (452415.91032238252, 206418.74768511369),
    2570: (452415.68718924839, 207335.36782698822),
    2572: (449663.33610940503, 209167.13277178022),
    2573: (449665.39498988277, 210084.39421717002),
    2574: (449664.42323207163, 211001.82726484918),
    2575: (450582.29631623952, 208251.89590644254),
    2576: (450581.9053864188, 209169.67328096708),
    2577: (450583.29205249622, 210086.52104155198),
    2583: (452414.4291987372, 208251.72387988595),
    2584: (452415.09925185668, 209169.53539625759),
    2588: (453333.46479139361, 201834.4451865858),
    2589: (453331.95987896924, 202751.40410128961),
    2590: (453332.13594133727, 203668.53383136354),
    2593: (454247.50073518371, 202751.23971250441),
    2594: (454248.12170020386, 203668.51922434368),
    2597: (455165.54771764105, 202751.03839555787),
    2598: (455165.47211411892, 203669.19776463267),
    2603: (453331.21754538687, 204585.5174666637),
    2604: (453331.98750959791, 205502.05752572144),
    2605: (453331.24806580396, 206417.68972628826),
    2606: (453332.1148533299, 207336.05405913451),
    2607: (454249.16141248826, 204585.23652909664),
    2608: (454248.23316329095, 205502.58158632374),
    2609: (454248.31319587171, 206418.29606150393),
    2610: (454248.53242847562, 207334.54058771511),
    2612: (455166.59495257848, 205501.09410160859),
    2613: (455166.73990575952, 206417.83877819678),
    2614: (455163.93216277892, 207334.39865799341),
    2618: (456080.67203641078, 207334.0465145215),
    2619: (453333.25990631629, 208252.02178324395),
    2623: (454248.23726063699, 208251.90292474729),
    2627: (455165.28811999154, 208252.35041511542),
    2631: (456082.48076873791, 208252.31447046637),
    2733: (459748.48375305417, 189000.38567863638),
    2760: (462499.33169628482, 184416.6509955432),
    2761: (462498.71278643585, 185335.01518757801),
    2762: (463414.76846406935, 182586.06825647893),
    2769: (460666.45730665681, 189003.07510846868),
    2770: (461582.15014206048, 186251.43149620973),
    2772: (461583.17539895047, 188083.54467407736),
    2785: (464333.41855637298, 181667.60420131753),
    2795: (467079.15163364826, 179833.99098493406),
    2798: (464332.1937042338, 182585.51349055913),
    2844: (459748.72631959885, 191750.65753935973),
    2847: (456997.5453540723, 194500.33944191789),
    2848: (456999.14465004159, 195420.23667532994),
    2852: (457916.38047195785, 195419.96546529085),
    2854: (458831.49276305328, 193585.87604164047),
    2862: (457000.69281007705, 197252.53583802056),
    2878: (460665.66056860518, 189915.02235057726),
    2880: (460665.57930458471, 191750.16239531059),
    2882: (461581.82949289313, 189919.59644253476),
    2883: (461583.02792168478, 190835.35835896427),
    2884: (461581.33536609553, 191751.06046902022),
    2885: (461580.22909733956, 192669.40825026957),
    2887: (462497.66216840595, 190833.82408314376),
    2892: (463416.08698396623, 191750.39838604111),
    2993: (456998.56464994745, 207334.51643302356),
    2997: (457917.34275805362, 207333.4961082915)
}
scale = (2.2004998202088553, 2.1987468516915558)

These offsets were calculated by:

• For each stop, finding the absolute position as given by the NaPTAN database
• Instantiating two variables, ∑∆offset and ∑∆position, to null 2D-vectors
• For each pair of stops appearing on the same map, adding the positive differences of their offsets within the map, and their absolute positions, to the respective variables
• Dividing ∑∆offset by ∑∆position to give a conversion factor between pixels and metres (given above as scale)
• For each map finding the average of each stop’s location minus its offset multiplied by the conversion factor (given above in map_numbers)

Here’s a bit of Python to convert between a map number and x and y offsets, and the WGS84 co-ordinate system. I’ve cheated a little in using Django’s GIS functionality which in turn uses the ctypes module to call functions from the GEOS library. If you’re not using Python or don’t want such a large dependency then you may wish to read the documentation linked from this page on the Ordnance Survey website.

from django.contrib.gis.geos import Point

def to_wgs84(map_number, rel_pos):
    """
    Takes an ACIS map number and a two-tuple specifying the offset on that
    map. Returns a Point object under the WGS84 projection.
    """
    corner = map_numbers[map_number]
    # Differing signs as we're applying a left-down offset to a left-up position
    pos = (
        corner[0] + rel_pos[0] * scale[0],
        corner[1] - rel_pos[1] * scale[1],
    )

    # 27700 is BNG; 4326 is WGS84
    return Point(pos, srid=27700).transform(4326, clone=True)

def from_wgs84(point):
    """
    Takes a Point object under any projection and returns a map number and
    two-tuple for that point. Raises ValueError if the point does not lie on
    any maps we know about.
    """
    # Make sure we're using the British National Grid
    pos = point.transform(27700, clone=True)
    for map_number, corner in map_numbers.items():
        rel = (
            (pos[0] - corner[0]) / scale[0], 
            (corner[0] - pos[1]) / scale[1],
        )
        # This is the right map if it appears in the 418 pixel square to the
        # lower-right of the the corner.
        if 0 <= rel_pos[0] < 418 and 0 <= rel_pos[1] < 418:
            return map_number, rel_pos
    raise ValueError("Appears on unknown map")

Next steps

The terms of use for the OxonTime website forbid using it for other than personal non-commercial purposes, making it an abuse of the terms to use this data in some sort of mash-up. From my reading of the terms, however, there’s nothing to stop one writing and distributing a client that uses this API directly. If you’ve got the time and inclination, why not write an iPhone/Android/mobile-du-jour client application using all sorts of fancy geolocation and free mapping data?

You could even scrape the real-time information from http://www.oxontime.com/pip/stop.asp?naptan=naptan-code&textonly=1 (just be wary about non-well-formed HTML). Obviously, make yourself aware of the terms, and if in doubt, contact Oxfordshire Country Council. Be warned that this API, though likely stable, comes with no guarantee to that effect. Also, it seems a little slow at times, so be gentle and treat it with respect.

Slides from a presentation to ‘The Alternative Guide’ JISC Project Assembly – 9th December 2009

Download now or preview on posterous

Mobile Oxford & Erewhon – The Alternative Guide UCLan.pdf (11197 KB)

Posted via email from Erewhon’s Posterous

Erewhon at Handheld Learning Conference

Posted via email from Erewhon’s Posterous

Mobile Oxford & Erewhon @ FOTE 2009

I’ll be joining Peter Robinson and Carl Marshall (from our sister http://steeple.posterous.comSteeple podcasting project) at the Future of Technology in Education conference today in London. 

The Future of Technology in Education Conference 2009 (FOTE09) is dedicated to showcasing the hottest technology related trends and challenges impacting the academic sector over the next 1 – 3 years and builds on the success of our inaugural event in 2008.

Date:      Friday, October 2nd, 2009

Venue:    Royal Geographic Society, Exhibition Road, London

The 2008 conference completely exceeded our expectations and we were taken back with the great feedback we received for bringing together a diverse mix of speakers to give an insight into the unique technology related challenges currently facing the academic sector.

via fote-conference.com

Posted via email from Erewhon’s Posterous

M&S Using Datamatrix 2D Bar Codes

Alex Dutton and I are on our way to see the guys in Bristol who are
part of the JISC funded Rapid Innovation project “Mobile Campus
Assistant” (link to follow).

En route we noticed that Marks and Spencer have started putting 2D
barcides on some of their products to provide bits of information to
their customers.

Although there isn’t all that much useful on there at the moment, I
think it has potential to deliver some interesting ideas in the
future. I would be quite interested to see what kind of take up M&S
have from this.

My personal incling is that there may well be a large response
initially as they publish these codes on their products but unless
they can deliver some compelling content the trend will die down.

See and download the full gallery on posterous

Posted via email from Erewhon’s Posterous

Datamatrix (2D Barcode) for m.ox.ac.uk

Points your Datamatrix readers at this! :) (Link is due to be live on the
5th October).

Posted via email from Mobile Oxford

Talking at the Open University Library Seminar

Yesterday Sebastian and I spoke at the Open University library seminar
regarding OxPoints and the upcoming Mobile Oxford (m.ox.ac.uk – coming
soon) website for mobile devices.

Slides below!

Download now or preview on posterous

intro.pdf (1664 KB)

Download now or preview on posterous

geo.pdf (637 KB)

Download now or preview on posterous

Erewhon – Open University Library Seminar 24th September.pdf (4510 KB)

Posted via email from Erewhon’s Posterous

QR (2D Barcode) for m.ox.ac.uk

Points your QR Code readers at this! :) (Link is due to be live on the
5th October).

Posted via email from Mobile Oxford

Microsoft’s Sense Cam [via Techcrunch]

A device that you wear around your neck, photographs, geo-locates,
auto-tags and then automatically uploads your life to the cloud?
That’s what Microsoft Research at Cambridge came up with some time ago
with the “Sensecam”. Techcrunch makes a good point in saying that
although the concept of wearing such a device may seem rather horrific
to many people today, people of yesteryear would have been
uncomfortable making their personal photo albums available for the
world to see as we do with places such as Facebook and Flickr.
 
I think the concept is certainly pertinent and now is almost
inevitable, as a serious photographer I carry a GPS logger with me
whenever I go taking photographs and whenever I travel great distances
I like to keep a log of where I went for future reference. I would
quite like this information available to my ancestors to come too, so
a device like the sense cam would only make recording it easier.
 
Of course recording every minute of your day would certainly involve
storing a lot of pointless data, so you would need tools to help sift
through the data. One of the features the Sensecam concept
incorporated was that of ambient sensors to trigger recording only
when significant events happened, e.g. a change in lighting,
temperature or movement.
 
[Via TechCrunch]

Posted via email from Erewhon’s Posterous