Category Archives: Python

Namepy – on the shoulders of giants

Whilst my core skill/tool is Python, I’m always learning new things, either inside or outside the Python ecosystem. I recently had the pleasure of working with Angular and Python/Flask. Here is a playful application based on these, plus Highcharts.

Going through “Python for Data Analysis”, some of the examples use a database of frequency of (US) baby names since 1880. I thought I’d combine this with a bit of Scrabble™.

In the Python world it’s common to add “py” to a word when making up names, so I’m calling this project “namepy”.

Since I’ll be using various frameworks and libraries, all created by others, I’ve subtitled this “On the shoulders of giants”.

Taking small steps often results in faster progress, so that’s what I’m be doing here.

Technical set up

The source code is at https://github.com/CoachCoen/namepy, with one branch per step.

Many production sites Content Delivery Networks for serving Javascript frameworks and libraries, usually minified, which helps to take the load of the server and may speed up first page load. To keep things simple and stable over time, I’m using full-sized, downloaded, copies.

I’m using WebFaction (affiliate link) as the host, since they make it easy to create Flask, Django and similar projects. And, as a popular host for developers, you’ll find lots of helpful documentation for developers online.

Getting started

Create a project folder

mkdir namepy
cd namepy

At the start of each of the steps

cd (my folder for personal projects)
cd namepy
git clone https://github.com/CoachCoen/namepy.git -b step1 step1

Note: “-b step1” specifies the name of the branch to clone. The second “step1” is the target folder, i.e. namepy/step1.

Next

Continue to Step 1 – Angular “Hello World”

Investment Tracking System – Django/Python

My client, a start up with a lot of experience in their field, had identified an important gap in the market. Large sums of money were being invested, with very long payback periods, without access to effective performance tracking tools.

They designed a tool to cover the gap and asked me to create a demonstration system in preparation for generating interest and raising capital.

I developed the system in Django, Python, PostgreSQL and Javascript. The front end uses a dashboard template based on Bootstrap and jQuery. Graphs are created using the excellent Highcharts charting library.

The resulting system imports the base data and generates monthly cost and revenue forecasts, taking into account seasonal variations, tax allowances and more.

Selection_005

The main management screen gives quick access to some key performance indicators.

Selection_006

Constraints can be defined, and potential investments can be checked against them.

Selection_007

Actual results can be compared against the projections.

Selection_008

Different heat maps show absolute or relative performance by state or county.

Selection_009

This was an eight month intensive project, resulting in a demo site which generated a lot of interest in the industry and allowed the client to achieve their first round of funding.

Installing Python 2.10 on Linux Mint 17.2

Warning: You may have some packages on your machine which rely on Python, and which may no longer work after installing a different version of Python. Ideally you should use a virtual machine for this

From https://slobaexpert.wordpress.com/2015/07/09/upgrade-python-2-7-6-to-python-2-7-10-on-linux-mint-os/:

  1. Download Gzipped source tarball from the Python website
  2. Unzip the downloaded file
  3. Switch to new folder (e.g. /Downloads/Python-2.7.10)
  4. In a terminal window:
    1. apt-get install libc-dev
      1. If you miss this, you’ll get an error: error: C compiler cannot create executables
    2. ./configure
    3. sudo make install
  5. Run “python –version”
    1. You should now see 2.7.10

Bottle – Python micro framework

Like Flask, Bottle is a Python micro-framework. It is so micro that it only consists of a single file. Whilst Flask is already a fairly small framework, some developers prefer Bottle, mainly for its easy of setup (single file, no dependencies)

The Bottle home page gives a Hello World example

Local trial

  1. Create and activate a new virtualenv or use a virtual machine.
    1. I use a VM for experimenting with Python libraries and frameworks, to keep it separate from my client work
  2. sudo pip install bottle
  3. Create hello_world.py with the code from the Bottle home page
  4. Run the script: python hello_world.py
  5. Check in your browser: http://localhost:8080/hello/world
    1. This should show “Hello world!”
    2. Also try it with your own name, e.g. http://localhost:8080/hello/Coen

On a server – using wsgi

For Python I use WebFaction, which makes it very easy to create new Python applications

In your WebFaction control panel:

  1. Domains / Websites -> Websites
    1. Select the domain
  2. Click on “Add an application” -> Create a new application
    1. Name: bottle_hello_world
    2. Category: mod_wsgi
    3. Type: mod_wsgi / Python 2.7 (note: Bottle also works with Python 3.x)
    4. URL: /bottle-hello-world
    5. Save
    6. Click Save again
  3. ssh into the host
    1. cd webapps
    2. cd <app name> (e.g. bottle_hello_world)
    3. cd htdocs
    4. pip install bottle
  4. Check bottle is installed
    1. python2.7
    2. import bottle
  5. Adapt code for wsgi (based on instructions at http://bottlepy.org/docs/dev/deployment.html#apache-mod-wsgi) and replace the contents of index.py with:

    import os
    # Change working directory so relative paths (and template lookup) work again
    os.chdir(os.path.dirname(__file__))

    import bottle
    from bottle import route, run, template
    application = bottle.default_app()

    @route(‘/hello/<name>’)
    def index(name):
    return template(‘<b>Hello {{name}}</b>!’, name=name)

  6. Test it, e.g. http://cm-demo.com/bottle-hello-world/index.py/hello/fred
    1. This should show “Hello fred!”

 

A simple Python Flask app on WebFaction

Flask is a popular Python-based micro framework. Here is how to install it on WebFaction.

This is based on the Flask instructions and the WebFaction instructions.

  1. Log into your WebFaction control panel
  2. Domains/Websites -> Applications -> Add new application
    1. Name: FlaskTest
    2. mod_wsgi
    3. mod_wsgi / Python 2.7
    4. (keep port closed)
  3. Domains/Websites -> Websites
    1. Click on website
    2. Add an application -> Reuse an existing application
    3. Select flasktest, url: http://cm-demo.com/flasktest
    4. Save the Website (important: this step is easy to miss, I’ve missed it myself a few times)
  4. ssh into host
    1. cd webapps/flasktest
    2. vi setup.sh
    3. cut and paste script from https://community.webfaction.com/questions/12718/installing-flask
    4. Change APPNAME and URLPATH (e.g. /foo)
    5. exit vi (<esc>ZZ)
    6. sh setup.sh
  5. Test it: point your browser to <domain>/foo
    1. This should show “Hello World!”
    2. Edit webapps/flasktest/flasktest/__index__.py, line:
      return “Hello World!”
      to return some different text
    3. Refresh page in browser
    4. Browser should now show the new text
  6. Set up the static files – make all files in /foo/static static:
    1. Domains/Websites -> Websites
    2. Click on website
    3. Add an application
      1. Create a new application
      2. Name: flasktest_static
      3. Symbolic link
      4. Symbolic link to static-only app
      5. Extra info: /home/<user_name>/<webapps/<app>/<app>/static
    4. Create the static folder
    5. Create a test page in the new static folder
    6. Try loading the page in your browser

Python for Data Analysis and Natural Language Processing

As I’m making my way through Natural Language Processing with Python and Data Science from Scratch: First Principles with Python, the first step is to set up the development environment.

My first attempt was to install numpy, python, nltk, matplotlib, IPython, etc, one at a time. However, I hit a few clashes between Python versions, so switched to Anaconda instead:

  1. Download Anaconda
  2. From the download folder, execute
    1. sh <name of downloaded file>
    2. Accept the defaults, but answer yes to preprend to PATH
  3. To check it works, start Python
    1. import numpy
    2. import pandas
    3. import matplotlib
  4. Download the nltk assets. Start Python and enter:
    1. import nltk
    2. nltk.download()
    3. use GUI to download “all”

Python, Bottle and websockets

Here is a simple websockets demo, using Python and Bottle, based on the example on the Bottle website

  1. Requirements:
    • pip install bottle
    • pip install python-dev
    • pip install gevent-websocket
  2. Create websockets.py, from the source code at http://bottlepy.org/docs/dev/async.html#finally-websockets
  3. python websockets.py
    • This starts the (websockets) server. Note that it doesn’t show any output
  4. Create websockets.html, again from http://bottlepy.org/docs/dev/async.html#finally-websockets
    • Make sure to change the websocket address (from “ws://example.com:8080/websocket”). On your local machine this should be “ws://localhost:8080/websocket”
  5. Load websockets.html in your browser (e.g. as a local file, at file:/// etc)
    • This should come up with an alert saying: Your message was ‘Hello, world’

Simple websocket client in Python

If you’re developing a websocket app you may need to throw some tests at it. Whilst looking for a websocket client I came across this client,  which makes it very easy to generate websocket calls

To test it out:

    1. pip install websocket-client
    2. python
      1. import websocket
      2. ws = websocket.WebSocket()
      3. ws.connect(“ws://example.com/websocket”)
      4. ws.send(“Hello, World”)

Python, CherryPy and Websockets – hosted

To set up websockets on a host can take a few extra steps. The host needs to know what to do with the websocket traffic.

It took a few attempts to get a Python/websockets sample up and running. The first websocket library I used (for Python/Bottle) worked very well on my local machine, but the host didn’t have the required libraries installed and didn’t give me enough access rights to fix this. I switched to a different framework and library, which worked much better.

  1. Create a new (WebFaction) application, wstest, custom websocket
    1. Save it, attach it to a domain/website, and make a note of the port
  2. ssh into the host
  3. Set up the requirements:
    1. pip install cherrypy
    2. pip install ws4py
    3. pip install argparse
  4. Switch to the application’s folder (cd webapps/wstest)
  5. Create index.html and site.py as created by Sylvain Hellegouarch, but with following changes:
    ….
    return index_page {d34bf16ac7b745ad0d2811187511ec8954163ba9b5dbe9639d7e21cc4b3adbdb} {‘username’: “User{d34bf16ac7b745ad0d2811187511ec8954163ba9b5dbe9639d7e21cc4b3adbdb}d” {d34bf16ac7b745ad0d2811187511ec8954163ba9b5dbe9639d7e21cc4b3adbdb} random.randint(50, 1000),
    ‘ws_addr’: ‘ws://<url>/ws’}
    ….
    cherrypy.config.update({
    ‘server.socket_host’: ‘127.0.0.1’,
    ‘server.socket_port’: <the application’s port – see above>
    })

    1. Important (this took me a while to figure out): the original websocket url (first/second line, above) in the sample code is “ws://localhost:9000/ws”. Having tried a bunch of different ports (80, 9000, the application’s port, etc), the correct solution is to leave the port off completely, so just use “ws://<url>/ws” instead
  6. Start CherryPy: python2.7 site.py
    1. Since you’re running this still attached to your terminal session, when you close the (ssh) terminal the process will stop as well.
  7. Now open the url in two separate browsers. You should be able to enter messages in either browser and see it appearing in both