Scraping www data with Python

On reading about the Revoke article 50 ePetition at the weekend and as its a hot topic at the moment, I thought I would write something to pull the data and breakdown where the signatures were coming from.

The petition data is available as a JSON file.

I was going to do this in PERL, something I have user for years, before I thought it might be a good exercise to try in Python.

So attempt one, looked a bit like this, using Python 3.7.1 on OSX.

Fetch the JSON file from the Internet
Decode it to a Dict data structure
pull out the values of interest and display

from io import StringIO
import json
import urllib.request

url = "https://petition.parliament.uk/petitions/241584.json"
pageObject = urllib.request.urlopen( url )

pageContentAsString = pageObject.read().decode(pageObject.headers.get_content_charset())

decodeJson = json.load(StringIO(pageContentAsString))
#print (json.dumps(decodeJson, sort_keys=True, indent=4))

print("Total: " + str(decodeJson["data"]["attributes"]["signature_count"]))
for country in decodeJson["data"]["attributes"]["signatures_by_country"] :
	if(country["name"] == "United Kingdom"):
		print("UK   : " + str(country["signature_count"]))

That was it. However later that evening, I went back to clean up the output formatting.

Format the output to be clearer
- display the top 20 countries with % of signatures
- display the top/bottom 20 constituencies
Write the downloaded JSON to a file.
- Use DateTime to datestamp filename,

That got me to the following output. Shown here are signatures from the top 20 countries, then the top 20 constituencies followed by the bottom 20 constituencies for signatures.

Write downloaded JSON to epetitions/brexit/190325_2329.json

DateTime : 2019-03-25 23:29:41.409749

Total :  5,648,486 United Kingdom                    5,416,903 95.90%
Total :  5,648,486 France                               41,783 0.74%
Total :  5,648,486 Spain                                23,763 0.42%
Total :  5,648,486 United States                        22,858 0.40%
Total :  5,648,486 Germany                              19,225 0.34%
Total :  5,648,486 Australia                            17,538 0.31%
Total :  5,648,486 Canada                               10,163 0.18%
Total :  5,648,486 Netherlands                           9,275 0.16%
Total :  5,648,486 Ireland                               8,994 0.16%
Total :  5,648,486 New Zealand                           6,536 0.12%
Total :  5,648,486 Switzerland                           5,994 0.11%
Total :  5,648,486 Italy                                 5,477 0.10%
Total :  5,648,486 Belgium                               4,764 0.08%
Total :  5,648,486 Gibraltar                             4,369 0.08%
Total :  5,648,486 Sweden                                3,680 0.07%
Total :  5,648,486 Hong Kong                             2,922 0.05%
Total :  5,648,486 Denmark                               2,666 0.05%
Total :  5,648,486 Austria                               2,605 0.05%
Total :  5,648,486 Portugal                              2,596 0.05%
Total :  5,648,486 United Arab Emirates                  2,387 0.04%

34900    Thangam Debbonaire MP          Bristol West
30320    Catherine West MP              Hornsey and Wood Green
27399    Caroline Lucas MP              Brighton, Pavilion
27335    Tulip Siddiq MP                Hampstead and Kilburn
26488    Rt Hon Keir Starmer MP         Holborn and St Pancras
26462    Rt Hon Diane Abbott MP         Hackney North and Stoke Newington
26312    Daniel Zeichner MP             Cambridge
26203    Deidre Brock MP                Edinburgh North and Leith
26104    Zac Goldsmith MP               Richmond Park
25582    Rt Hon Jeremy Corbyn MP        Islington North
25344    Meg Hillier MP                 Hackney South and Shoreditch
25029    Helen Hayes MP                 Dulwich and West Norwood
23748    Neil Coyle MP                  Bermondsey and Old Southwark
23721    Kate Hoey MP                   Vauxhall
23408    Chuka Umunna MP                Streatham
23314    Vicky Foxcroft MP              Lewisham, Deptford
23312    Rt Hon Mark Field MP           Cities of London and Westminster
23302    Rt Hon Sir Vince Cable MP      Twickenham
23188    Marsha De Cordova MP           Battersea
23105    Rushanara Ali MP               Bethnal Green and Bow

1816     Eddie Hughes MP                Walsall North
1972     Ian Austin MP                  Dudley North
1974     Rt Hon Pat McFadden MP         Wolverhampton South East
1980     Mr Adrian Bailey MP            West Bromwich West
2079     Grahame Morris MP              Easington
2083     Nick Smith MP                  Blaenau Gwent
2154     Emma Reynolds MP               Wolverhampton North East
2188     Karl Turner MP                 Kingston upon Hull East
2196     Tom Watson MP                  West Bromwich East
2221     Melanie Onn MP                 Great Grimsby
2232     Angus Brendan MacNeil MP       Na h-Eileanan an Iar
2237     Sarah Champion MP              Rotherham
2279     Stephanie Peacock MP           Barnsley East
2341     Chris Bryant MP                Rhondda
2346     Mike Wood MP                   Dudley South
2399     Rt Hon Liam Byrne MP           Birmingham, Hodge Hill
2400     Jack Brereton MP               Stoke-on-Trent South
2438     Gordon Marsden MP              Blackpool South
2438     Ruth Smeeth MP                 Stoke-on-Trent North
2472     Anna Turley MP                 Redcar

The next day, I thought of my Raspberry PI. I added a summary file append, with just the UK data and after installing the modules required, it was set running every 15 minutes using a cron job.

## crontab -l gives, 
*/15 * * * * /usr/bin/python3 /home/pi/ripPetition.py

Over time this gave the following output

2019-03-24 23:30:03.764265,5305289,United Kingdom,5093045,96.00
2019-03-24 23:45:03.690786,5308423,United Kingdom,5095988,96.00
2019-03-25 00:15:04.629576,5313114,United Kingdom,5100484,96.00
2019-03-25 00:30:04.227742,5314813,United Kingdom,5101958,96.00
2019-03-25 00:45:03.546101,5316210,United Kingdom,5103097,95.99
2019-03-25 01:00:03.889650,5317601,United Kingdom,5104143,95.99
2019-03-25 01:15:04.158192,5318623,United Kingdom,5105560,95.99
2019-03-25 01:30:03.638560,5319530,United Kingdom,5106118,95.99
2019-03-25 01:45:03.957981,5320350,United Kingdom,5106118,95.97
2019-03-25 02:00:04.410973,5321054,United Kingdom,5107640,95.99
2019-03-25 02:15:04.364589,5321701,United Kingdom,5107809,95.98
2019-03-25 02:30:03.899075,5322303,United Kingdom,5108611,95.98
2019-03-25 02:45:04.169249,5322894,United Kingdom,5109133,95.98
2019-03-25 03:00:04.066469,5323416,United Kingdom,5109337,95.98
2019-03-25 03:15:03.744909,5323877,United Kingdom,5109863,95.98
2019-03-25 03:30:03.662507,5324376,United Kingdom,5110317,95.98
2019-03-25 03:45:04.128469,5324818,United Kingdom,5110669,95.98
2019-03-25 04:00:03.608843,5325278,United Kingdom,5111061,95.98
2019-03-25 04:15:04.069418,5325733,United Kingdom,5111267,95.97
2019-03-25 04:30:04.025370,5326172,United Kingdom,5111684,95.97

To visualise the data, I used Matplotlib with a separate Python script. It is available here along with the code above. The following chart is the output from this script. I think I even found a bug. I did email the ePetitions site about this 🙂 but didn’t hear back.

Overall what started as a quick experiment, became me learning a bit more Python and a fun use of a Raspberry PI. Thanks to the ePetitions website, the BBC and the various StackOverflow posts that helped me out, especially with the x-axis ticks.

The code can be found here.