Category Archives: Flask

Namepy step 7 – Bringing it all together

(This is part of the namepy project. Start at Namepy – on the shoulders of giants)

Time to show some real results on a web page.

  1. Extend the API to show the letter scoring tables, no pagination, in __init__.py add:
    manager.create_api(models.Set, methods=['GET'], results_per_page=0) 
    
  2. Rename helloworld.html to index.html
  3. At the end of views.py, update the template name to index.html, and stop passing in ‘names’ since this is now done through the API, and rename the endpoint function to ‘index’:
    @app.route("/") 
    def index(): 
        return render_template('index.html') 
    

That’s it for the changes to the back end. The rest of the changes will all be in the front end, in index.html

  1. Rename the app from HelloWorldApp to NamePyApp
  2. Rename the controller from HelloWorldController to NamePyController
  3. Load the letter scoring table, and simplify it for faster lookup
    $scope.sets = [];
    angular.forEach(response.data.objects, function(set, index) {
        scores = {};
        angular.forEach(set.scores, function(score, index) {
            scores[score.letter] = score.score;
        });
        $scope.sets.push({ name: set.name, scores: scores});
    });
    
  4. Calculate the score for each of the sets
    angular.forEach($scope.sets, function(set, index) {
        var total = 0;
        var error = false;
        angular.forEach(name.split(''), function(character, index2) {
            if (character in set.scores) {
                total += set.scores[character];
            } else {
                error = true;
            }
        });
    
        if (error == false) {
            result.push([set.name, total]);
        }
    
        $scope.sort_on_element(result, 1);
    
        $scope.scores = result;
    });
    
  5. Show the result on the page, using Highcharts. For the code see the source code, function “showLetterScores”

Show baby name distribution

  1. Get data for entered name
    var filters = [{ name: 'name', 
        op: 'ilike', 
        val: $scope.visitor_name}];
    
    $http({
        method: 'GET',
        url: 'api/name',
        params: {"q": JSON.stringify({"filters": filters})}
        })
        .then(
            $scope.show_name_distribution,  
            function(response) {            
                $('#babynames_container').hide();
            }
        );
    
  2. Restructure the results for Highcharts
    var boy_frequency = [];
    var girl_frequency = [];
    var boys_found = false;
    var girls_found = false;
    
    angular.forEach(response.data.objects[0].frequencies, 
        function(frequency) {
            boy_frequency.push([
                Date.UTC(frequency.year, 1, 1), 
                frequency.boys_count]);
    
            girl_frequency.push([ 
                Date.UTC(frequency.year, 1, 1), 
                frequency.girls_count]);
    
            if (frequency.boys_count) boys_found = true;
            if (frequency.girls_count) girls_found = true;
        });
    
    $scope.sort_on_element(boy_frequency, 0);
    $scope.sort_on_element(girl_frequency, 0);
    
  3. Show the results using Highcharts. See the source code, function “show_name_distribution”

Done

Done Done

This is the final blog post for this little project. I hope you found it useful.

Namepy step 6 – Load the data into the database

(This is part of the namepy project. Start at Namepy – on the shoulders of giants)

We will need the following data in the database:

  • Name frequencies – baby names by year
  • Scrabble™ letter values, by (country) Scrabble™ set

Name frequencies

  1. Download the data from https://www.ssa.gov/oact/babynames/names.zip
  2. Unzip it in <project_root>/raw_data/yob1880.txt, etc
  3. You may want to add raw_data to .gitignore, so it doesn’t get stored in your git repo
  4. Create some code to read files and store in PostgreSQL – read_name_frequencies.py
    def read_frequencies_from_file(filename, names):
        print(filename)
        year = int(filename[3:7])
    
        year_frequencies = {}
        for name in names:
            year_frequencies[name] = {'F': 0, 'M': 0}
    
        with open('raw_data/%s' % filename) as file:
            for line in file.readlines():
                try:
                    name_text, sex, count = line.split(",")
                except:
                    print("Couldn't parse line")
                    print(line)
                    print
                    continue
    
                if name_text not in names:
                    name = Name(name=name_text)
                    db.session.add(name)
                    db.session.commit()
                    names[name_text] = name.id
                    year_frequencies[name_text] = {'F': 0, 'M': 0}
    
                year_frequencies[name_text][sex] = int(count)
    
            for name, name_frequency in year_frequencies.iteritems():
                if name_frequency['F'] + name_frequency['M']:
                    name_id = names[name]
                    frequency_record = NameFrequency(name_id=name_id,
                        year=year,
                        boys_count=name_frequency['M'],
                        girls_count=name_frequency['F'])
                    db.session.add(frequency_record)
                    db.session.commit()
    
    def read_name_frequencies():
        db.create_all()
    
        # Start with an empty list
        print("Deleting any previous data")
        db.session.query(NameFrequency).delete()
        db.session.query(Name).delete()
        db.session.commit()
        print("Done")
    
        names = {}
    
        # Get file list
        for filename in listdir('raw_data'):
            if filename[:3] == 'yob':
                read_frequencies_from_file(filename, names)
    
  5. Run the code. Note that this may take a while to run. On my development machine it took about 8 minutes
  6. Check this in the database, for instance with phpPgAdmin or pgAdmin

Scrabble™ letter values

For a list of Scrabble™ letter values by Scrabble™ set see this Wikipedia entry. The following code will grab this page, extract the letter values and save this in the database

  1. Add the new tables to models.py
    class Set(db.Model):
        id = db.Column(db.Integer, primary_key=True)
        name = db.Column(db.String())
        scores = db.relationship('LetterScore', backref='set', lazy='dynamic')
    
    class LetterScore(db.Model):
        id = db.Column(db.Integer, primary_key=True)
        set_id = db.Column(db.Integer, db.ForeignKey('set.id'))
        score = db.Column(db.Integer)
        letter = db.Column(db.String(1))
    
  2. Write some code to parse this page and store the results in the database. See the source code in my GitHub repo
  3. Run the code
  4. Check the results in the database. See above (end of name frequencies section) for some suggested tools

Done

Next

Time to pull it all together and show some real charts

Continue to Step 7 – Bringing it all together

Namepy step 5 – Flask-Restless

(This is part of the namepy project. Start at Namepy – on the shoulders of giants)

We will need Angular to use an Ajax call to request the data from Flask, using a REST-style request, and show it in Highcharts.

The two main Flask libraries for creating a REST API are Flask-Restful and Flask-Restless. We will be using Flask-Restless, because it is particularly suited for what we’re trying to do: “Flask-Restless provides simple generation of ReSTful APIs for database models defined using SQLAlchemy (or Flask-SQLAlchemy)” – from the Flask-Restless documentation.

Create and test the REST API

  1. Install flask-restless
    (virtualenv) pip install flask-restless
  2. import at at the start of __init__py:
    import flask.ext.restless
  3. Create the API endpoint, add following to end of __init__.py:
    manager = flask.ext.restless.APIManager(app, flask_sqlalchemy_db=db) 
    manager.create_api(models.Name, methods=['GET']) 
    
  4. Test this – python index.py; point your browser to 127.0.0.1/api/name, this should show a JSON structure with the names and frequencies

Use Angular to request and process the REST data from the back end system

  1. Create a new function which takes a response object, extracts the json data, formats it, and passes it to Highcharts
                        $scope.showChart = function(response_data) {
    
                            chart_data = []
                            angular.forEach(response_data.objects, function(name_object, key) {
    
                                boys_count = []
                                angular.forEach(name_object.frequencies, function(frequency, key) {
                                    boys_count.push(frequency.boys_count);
                                });
                                chart_data.push({ name: name_object.name, data: boys_count });
                            });
    
                            $('#container').highcharts({
                                chart: {
                                    type: 'column'
                                },
                                title: {
                                    text: 'Name frequencies'
                                },
                                series: chart_data
                            });
                        };
    

    Note that this doesn’t quite make sense, for instance the year isn’t being shown in the chart. We’ll fix all of that later. For now the aim is to get the infrastructure set up – database, REST API, Angular, etc.

  2. Use Angular’s $http.get() function to call the api, and pass the response object to the showChart function upon completion
                        $http.get('api/name') 
                        	.then(function(response) { 
                        		$scope.showChart(response.data); 
                        	}); 
    
  3. Test: Make sure the Flask app is running and go to http://127.0.0.1:5000/. You should still see the name frequencies chart

Done

Next

That completes the technical set up, for now. We’re ready to do some real coding, starting with getting the data into the database.

Continue to Step 6 – Load the data into the database

Namepy step 4 – PostgreSQL, SQLAlchemy and Flask-SQLAlchemy

(This is part of the namepy project. Start at Namepy – on the shoulders of giants)

I’ve used PostgreSQL for a number of different projects. It is the database of choice for Django developers, with MySQL a close second. It is fast, stable, able to hand large databases and open source.

For a small project like this PostgreSQL is overkill, and SQLite would be easier to set up. However, I wanted to show how to integrate the key elements of a serious Flask/etc project, which has to include a production-grade database.

  1. Create a new database
    • For instructions on creating a new database in PostgreSQL you’ll find many helpful posts online, including my own blogpost
    • Make a note of the database name, user name and password
  2. Grab the required libraries (psycopg2 is needed for using flask-sqlalchemy with PostgreSQL, but not automatically installed with it)
    (virtualenv) pip install flask-sqlalchemy
    (virtualenv) pip install psycopg2
    
  3. Import flask-sqlalchemy
    from flask_sqlalchemy import SQLAlchemy
    
  4. Configure it. Replace <user name>, etc, in the text below with the actual user name, etc
    app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://<user name>:<password>@localhost/<database>'
    app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
    
  5. Define the models in <project_root>/models.py
    from hello import db
    
    class Name(db.Model):
        id = db.Column(db.Integer, primary_key=True)
        name = db.Column(db.String())
        frequencies = db.relationship('NameFrequency', backref='name', lazy='dynamic')
    
    class NameFrequency(db.Model):
        id = db.Column(db.Integer, primary_key=True)
        name_id = db.Column(db.Integer, db.ForeignKey('name.id'))
        year = db.Column(db.Integer)
        boys_count = db.Column(db.Integer)
        girls_count = db.Column(db.Integer)
    
  6. Test the database and create the tables
    python
    >>> from hello import db
    >>> db.create_all()
    

    As long as you see no error message this has worked

  7. Create a script for adding some test data. In hello.py add:
    def create_test_data():
        for name in ('Fred', 'Sue'):
            new_name = Name(name=name)
            db.session.add(new_name)
            for year in range(1990, 1996):
                new_frequency = NameFrequency(
                    name=new_name,
                    year=year,
                    boys_count=random.randint(50, 100),
                    girls_count=random.randint(50, 100))
        db.session.commit()
    
  8. And run the script
    python
    >>> from hello import create_test_data
    >>> create_test_data()
    
    
  9. Test this:
    >>> from models import Name
    >>> names = Name.query.all()
    >>> names[0].name
    u'Fred'
    >>>
    
    >>> names[0].frequencies[0].boys_count
    57
    >>>
    

    Or some other number between 50 and 100

Whilst this works so far, there are a couple of problems. The database details, including password, are in the main source code. And when you try running it as a Flask script (python hello.py) you get a circular import error. Let’s tidy this up

  1. Create a new file, <project_root>\config.py, containing the configuration (with the correct login and database details):
    app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://&lt;username&gt;:&lt;password&gt;@localhost/&lt;database name&gt;'
    app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
    
  2. If you’re using git, make sure this isn’t under version control. Create/update .gitignore to include the following:
    config.py
    
  3. Create a new file, create_test_data.py, containing create_test_data from hello.py, plus necessary imports, and the code to run the create_test_data function when called from the command line
  4. Create index.py:
    import os
    import sys
    
    CURRENT_FILE = os.path.abspath(__file__)
    CURRENT_DIR = os.path.dirname(CURRENT_FILE)
    PROJECT_DIR = os.path.dirname(CURRENT_DIR)
    
    sys.path.append(PROJECT_DIR + '/src/')
    sys.path.append(PROJECT_DIR + '/virtualenv/lib/python3.5/site-packages/')
    
    from app import app as application
    
    if __name__ == '__main__':
        application.run(debug=True)
    
  5. Create an “app” folder in the project_root folder
  6. In <project_root>/app create __init__.py:
    from flask import Flask
    from flask.ext.sqlalchemy import SQLAlchemy
    
    app = Flask(__name__)
    app.config.from_object('config')
    db = SQLAlchemy(app)
    
    from app import views, models
    
  7. Move models.py into the app folder and change the first line to:
    from app import db
    
  8. In the app folder, create views.py, containing @app.route(“/”) and hello() function from hello.py, plus necessary imports
  9. Test this. You should get

    Welcome, today is June 7th 2016

Let’s show some data

  1. In views.py, add:
    from models import Name
    
  2. In render_template, pass in the Name objects:
    return render_template('helloworld.html', names=Name.query.all())
    
  3. In the helloworld.html template, get the series data from the Name objects:
                        series: [
                            {% for name in names %}
                                {
                                    name: '{{ name.name }}',
                                    data: [
                                        {% for frequency in name.frequencies %}
                                            {{ frequency.boys_count }},
                                        {% endfor %}
                                    ]
                                },
                            {% endfor %}
                        ]
    
  4. The chart title no longer makes sense. Change this to “Name frequencies”
  5. Test this. You should see another chart

Done

Next

Continue to Step 5 – Flask-Restless

Namepy step 3 – Adding in Highcharts

(This is part of the namepy project. Start at Namepy – on the shoulders of giants)
Highcharts is a JavaScript library for creating high quality interactive charts. It is not open source but requires a license fee for commercial projects

  1. Grab Highcharts from http://www.highcharts.com/download
  2. Copy highcharts.src.js (for a production site use highcharts.js) into static/external
  3. Include this in helloworld.html:
     <script src="static/external/highcharts.src.js"></script> 
  4. Ditto for jQuery, download from https://jquery.com/download/, and include in helloworld.html, before the highcharts link
  5. Create a container for the chart:
    <div id="container" style="min-width: 310px; height: 400px; margin: 0 auto;"></div>
    
     
  6. Create a very simple chart, e.g.:
    			$(function () { 
    			    $('#container').highcharts({ 
    			        chart: { 
    			            type: 'column' 
    			        }, 
    			        title: { 
    		                text: 'Rabbit sightings' 
    			        }, 
    			        xAxis: { 
    			            categories: [ 
    			                'Spring', 
    			                'Summer', 
    			                'Autumn', 
    			                'Winter' 
    			            ], 
    			        }, 
    			        series: [{ 
    			            name: '2015', 
    			            data: [5, 7, 8, 3] 
    			        }, { 
    			            name: '2016', 
    			            data: [6, 6, 10, 5] 
    
    			        }] 
    			    }); 
    			}); 
    
  7. Start the Flask/python script (python hello.py) and view the result in your browser (127.0.0.1:5000)

 

Done

Next

Continue to Step 4 – PostgreSQL, SQLAlchemy and Flask-SQLAlchemy

Namepy step 2 – Flask and Angular

(This is part of the namepy project. Start at Namepy – on the shoulders of giants)

We’re going to need some server-side processing, for which I’ll be using Python and the Flask micro framework

  1. I recommend you use a virtualenv for this project
  2. Install flask
    pip install flask
    
  3. Create /hello.py
    from flask import Flask
    app = Flask(__name___)
    app.config['DEBUG'] = True
    
    @app.route("/")
    def hello():
        return "Hello world!"
    
    if __name__ == "__main__":
        app.run()
    

Test this:

  1. Start Flask and the hello.py Python script
    python hello.py
  2. In your browser, go to http://127.0.0.1:5000/. This should show

    “Hello world!”

Show the Angular page from step 1

  1. In hello.py, add
    from flask import render_template
  2. hello() function, replace last line with
    return render_template('helloworld.html')
  3. Move helloworld.html to (project root)/templates
  4. Test this (see above). This will bring up

    Welcome, today is {{ dateToday }}

As you can see, this isn’t quite right. If you open up the browser’s console (e.g. ctrl-shift-j in Google Chrome) you’ll see that the browser can’t find the external files (AngularJS and MomentJS). Let’s fix this first

  1. Create (project root)/static. This is where Flask will be looking for anything in (url)/static, without doing any further processing
  2. Move /external to /static. You should now have /static/external, containing angular.js and moment.js
  3. In templates/helloworld.html, change the link from “external/angular.js” to “static/external/angular.js”
  4. Do the same for moment.js
  5. Test this again. This will show

    Welcome, today is

Almost there. If you check in your console you’ll see that the error messages are no longer there. However, the {{ dateToday }} has disappeared. If you view the web page’s source code you’ll see that the line has been replaced with “<p>Welcome, today is </p>”. This is because the Jinja template engine used by Flask also uses double curly brackets {{ }}, just like Angular. Jinja processes (and removes) the double curly brackets before it gets to Angular. There are a number of solutions for this, such as telling Angular to use double square brackets [[ ]] instead:

  1. In helloworld.html, after the line “angular.module(‘HelloWorldApp’, [])”, add:
        .config(function($interpolateProvider){
            $interpolateProvider.startSymbol('[[').endSymbol(']]');
        })
    
  2. And replace the {{ }} brackets with [[ ]]:
    
    
    Welcome, today is [[ dateToday ]]
    
    
  3. Test this. You should now see

    Welcome, today is June 7th 2016

Going live

Done

Next

Continue to Step 3 – Adding in Highcharts

Namepy – on the shoulders of giants

Whilst my core skill/tool is Python, I’m always learning new things, either inside or outside the Python ecosystem. I recently had the pleasure of working with Angular and Python/Flask. Here is a playful application based on these, plus Highcharts.

Going through “Python for Data Analysis”, some of the examples use a database of frequency of (US) baby names since 1880. I thought I’d combine this with a bit of Scrabble™.

In the Python world it’s common to add “py” to a word when making up names, so I’m calling this project “namepy”.

Since I’ll be using various frameworks and libraries, all created by others, I’ve subtitled this “On the shoulders of giants”.

Taking small steps often results in faster progress, so that’s what I’m be doing here.

Technical set up

The source code is at https://github.com/CoachCoen/namepy, with one branch per step.

Many production sites Content Delivery Networks for serving Javascript frameworks and libraries, usually minified, which helps to take the load of the server and may speed up first page load. To keep things simple and stable over time, I’m using full-sized, downloaded, copies.

I’m using WebFaction (affiliate link) as the host, since they make it easy to create Flask, Django and similar projects. And, as a popular host for developers, you’ll find lots of helpful documentation for developers online.

Getting started

Create a project folder

mkdir namepy
cd namepy

At the start of each of the steps

cd (my folder for personal projects)
cd namepy
git clone https://github.com/CoachCoen/namepy.git -b step1 step1

Note: “-b step1” specifies the name of the branch to clone. The second “step1” is the target folder, i.e. namepy/step1.

Next

Continue to Step 1 – Angular “Hello World”

Flask and Angular on Heroku

I am working my way through this excellent tutorial, covering Python3, Flask, Angular, Heroku, SQLAlchemy, Alembic, requests, Beautiful Soup, NLTK, Redis and D3. Here are some extra notes

  • To stop me from blindly copying/pasting the code, I printed off the tutorial and worked from the paper version.
  • I had some problems installing Virtualenvwrapper (on Linux Mint 17.2), until I followed these instructions
  • I had some clashes with Anaconda
    • Virtualenvwrapper’s deactivate clashed with Anaconda’s deactivate. Prompted by these instructions I renamed ~/anaconda/bin/activate and ~/anaconda/bin/deactivate
    • “pip install psycopg2” resulted in:
      Error: “setuptools must be installed to install from a source distribution”
      After much experimentation I guessed that this might be due to Anaconda. I created a new virtual machine (without Anaconda) and re-started the tutorial. This fixed the psycopg2 problem

Part 1, set up Heroku

  • I used a free Heroku account. Between a dedicated server, a WebFaction account and a HotDrupal account I’m already paying enough for hosting
  • “heroku create wordcounts-pro” gave me an error “Name is already taken”. According to this Heroku page,  app names are in the global namespace, so I guess I’m not the first one to follow this tutorial. To work around this, I prepended the app name with my initials, i.e. “heroku create cdg-wordcounts-pro”, etc
  • So I can push the changes to heroku I set up public key access
  • Before running “git push stage/pro master”, make sure to check in the changes to git (git add, git commit)

Part 2, set up databases

  • To create the Postgres database:
    • sudo su — postgres
    • psql
      • # CREATE DATABASE wordcount_dev;
      • # CREATE USER ‘<your user name>’
      • # GRANT ALL PRIVILEGES ON wordcount_dev TO <your user name>;
  • After running “heroku run python manage.py db upgrade …” I got the error message:
    No such file or directory: ‘/app/migrations/versions’

    • Locally I had an empty directory <app folder>/migrations/versions. However, git ignores empty directories. This is why I could run “.. manage.py db upgrade” locally but not on heroku
    • Oops, I’d forgotten to run
      python manage.py db migrate
      Now it worked fine
    • If you make the same mistake, remember to propagate the changes to heroku and then re-run db migrate on heroku

Part 3, requests, Beautiful Soup and NLTK

  • At one stage I got a server error. To sort this I looked at the heroku log:
    heroku logs –app <heroku app name>
  • When I ran the nltk downloader I didn’t get the usual gui but a “tui” (text user interface). It was fairly simple to navigate, but I didn’t bother to specify the location of the tokenizers. Instead I used the default (~/nltk_data) and then moved nltk_data into my app folder
  • The links to Bootstrap and jQuery didn’t work, either because I mistyped them or because they are out of date. The Bootstrap and jQuery websites give you up-to-date CDN links, so use those instead

Part 4, Redis task queue

  • I used these instructions to install Redis on Linux Mint
  • Apart from the inevitable few typing mistakes, everything worked remarkably smoothly. Nothing else to add

Part 5, Adding in Angular

  • It all went well, until I added the getWordCount function to the controller. I’d put the function inside the main.js file, but outside of the controller. When poller got called, none of the dependencies were included, so it couldn’t find $http (first line of poller function)
    • The error was: $http not defined
    • Despite comparing my version with the author’s GitHub one, I couldn’t see the difference. In the end I used the author’s version (of main.js) instead of mine. That worked fine. It took another line by line comparison to find the problem
  • The word/frequency list is no longer sorted. jsonify loses the order

Part 6, Staging the changes, including Redis

  • So far I’ve been using a free account. When I tried to add on Redis, heroku tells me: Please verify your account to install this add-on plan (please enter a credit card)
    • If I understand it correctly, it is still free (but don’t take my word for it – and don’t come back to me if you end up getting charged)
    • I entered my credit card details for my Heroku again. Now I can add Redis
  • “heroku addons:add redistogo –app” gave a warning to say that “addons:add” has been deprecated.
    • I used “addons:create” instead

 

A simple Python Flask app on WebFaction

Flask is a popular Python-based micro framework. Here is how to install it on WebFaction.

This is based on the Flask instructions and the WebFaction instructions.

  1. Log into your WebFaction control panel
  2. Domains/Websites -> Applications -> Add new application
    1. Name: FlaskTest
    2. mod_wsgi
    3. mod_wsgi / Python 2.7
    4. (keep port closed)
  3. Domains/Websites -> Websites
    1. Click on website
    2. Add an application -> Reuse an existing application
    3. Select flasktest, url: http://cm-demo.com/flasktest
    4. Save the Website (important: this step is easy to miss, I’ve missed it myself a few times)
  4. ssh into host
    1. cd webapps/flasktest
    2. vi setup.sh
    3. cut and paste script from https://community.webfaction.com/questions/12718/installing-flask
    4. Change APPNAME and URLPATH (e.g. /foo)
    5. exit vi (<esc>ZZ)
    6. sh setup.sh
  5. Test it: point your browser to <domain>/foo
    1. This should show “Hello World!”
    2. Edit webapps/flasktest/flasktest/__index__.py, line:
      return “Hello World!”
      to return some different text
    3. Refresh page in browser
    4. Browser should now show the new text
  6. Set up the static files – make all files in /foo/static static:
    1. Domains/Websites -> Websites
    2. Click on website
    3. Add an application
      1. Create a new application
      2. Name: flasktest_static
      3. Symbolic link
      4. Symbolic link to static-only app
      5. Extra info: /home/<user_name>/<webapps/<app>/<app>/static
    4. Create the static folder
    5. Create a test page in the new static folder
    6. Try loading the page in your browser