Category Archives: Django

Django Rest Framework API Microservice

I recently completed a small project for Zenstores. They simplify the shipping process for ecommerce sites. Their online service lets online businesses use multiple shipping companies for deliveries.

Each shipping companies offers a different own API, for booking shipments, etc. My client uses a separate microservice for each shipping company. These microservices listen to requests from the main system and translate them to the shipping company’s standard.

My client asked me to use Django Rest Framework to create a microservice which supports a new shipping company. DRF is a popular and powerful library to create RESTful APIs using Django.

The supplier provided me with a sandbox API and extensive documentation. The documentation was somewhat incomplete and out of date. Fortunately their support contact was very helpful all along.

I used Test Driven Design for complex functions where I understood the functionality well. For the rest I used a more experimental approach and added unit tests afterwards. Testing coverage was over 90%.

The client has integrated the microservice within their system and the first test shipments have gone through.

Investment Tracking System – Django/Python

My client, a start up with a lot of experience in their field, had identified an important gap in the market. Large sums of money were being invested, with very long payback periods, without access to effective performance tracking tools.

They designed a tool to cover the gap and asked me to create a demonstration system in preparation for generating interest and raising capital.

I developed the system in Django, Python, PostgreSQL and Javascript. The front end uses a dashboard template based on Bootstrap and jQuery. Graphs are created using the excellent Highcharts charting library.

The resulting system imports the base data and generates monthly cost and revenue forecasts, taking into account seasonal variations, tax allowances and more.

Selection_005

The main management screen gives quick access to some key performance indicators.

Selection_006

Constraints can be defined, and potential investments can be checked against them.

Selection_007

Actual results can be compared against the projections.

Selection_008

Different heat maps show absolute or relative performance by state or county.

Selection_009

This was an eight month intensive project, resulting in a demo site which generated a lot of interest in the industry and allowed the client to achieve their first round of funding.

Linux package management: apt-get, aptitude, synaptic, wajig

When you set up your Linux system, you typically grab an image, burn it to DVD, boot the machine off the DVD and follow the instructions. The image will contain the most recent (stable) version of the core Linux distro files and of the packages which are bundled with it. The package managers let you keep your packages up to date, add new packages, remove packages, etc. They also deal with the dependencies; when installing a package, all packages it relies on are automatically installed as well

The package managers do this by accessing repositories (repos) of packages. Your distro came configured with a set of repos which contain packages which work on your repo. If you want to install some software which is not in your standard repos you may need to add a repo to the list

At the basic level, the system uses apt and dpkg to manage the software. The following are some front-ends on top of apt and dpkg:

  • apt-get: command line only
  • aptitude: very similar to apt-get, plus a text-based “gui”
  • synaptic: gui
  • wajig: slightly friendlier front-end to both apt-get and aptitude. Brings together the power of both

For more information have a look at this article

Django Migrations

Django has a great system for maintaining your database structure, for keeping your RDMS (typically MySQL, PostgreSQL or SQLite) in line with your Django-defined Models, and within your version control system.

I recently hit a snag with a live client site, which already has 100s of registered users, significant numbers in a small market. Something to handle with extreme caution. Time to dive a bit deeper into Django’s migrations.

Note: migrations were introduced in Django 1.7

Models

Instead of working directly on the database, Django expects you to define the database structure in Python code. This powers many of Django’s features. For instance, the admin system gives  administrators an easy way to maintain the data, and forms can be quickly created.

Once you’ve defined the models and configured the database connection (username, password, etc), Django creates the table structures for you.

Generate the initial database structure by running
>> python manage.py makemigrations

This generates the ‘migration’ path (see below) from an empty database to the initial database

Then apply this to the actual database
>> python manage.py migrate

If you’ve used Django 1.6 or earlier, note that syncdb has been deprecated and replaced by migrate

Migrations

But what happens when you change a table, e.g. when you delete or create a field? And how do you track the changes on your development machine, and make sure the same changes are applied to your test and live systems?

Every time you run
>> python manage.py makemigrations
Django compares the last known Model structure (by running through all past migrations) to the current Model structure and generates a new migrations file, one for each Django app with a changed Model structure. These files are stored in <app>/migrations and are numbered (starting with 0001).

Running
>> python manage.py migrate
applies the most recent migrations. If the database is empty, then Django will start from 0001_initial.py. Otherwise, it checks the migrations table to see, for each app, which migration is the latest one which has been applied to the database, and then runs any later migrations

Migrations and version control

Because the migration files are ordinary Python (text) files, stored in an app’s sub directory, your version control system will manage them for you. When you update the test or live system with the latest code changes, the latest migrations will also be copied across.

Just make sure to run
>> python manage.py migrate
after updating the source code through your version control system

 What went wrong

Normally Django’s migration system works beautifully, making sure the models and database structure stay in sync, between each other and across the development, test and live systems.

Due to some complicated reasons involving tight deadlines and budgets, holidays, and more, something went wrong with the migrations.

The client requested some extra database fields. I added the fields to the model and ran makemigrations.

First nothing happened. Just to be sure I ran
>> python manage.py makemigrations <app name>
and Django told me it had created migration 0001_initial, instead of an incremental change. Checking the database, no changes were made.

Diving into git, I could see a series of past migration files. I could probably roll back the changes to them (i.e. which deleted them), but decided to try something a bit simpler first.

 And how I fixed it

I deleted all existing migrations, from <app name>/migrations and commented out the new database fields in the model.

Running
>> python manage.py makemigrations
created a few 0001_initial migration file, which matches the original model and the current database structure

I removed the comments, to re-instate the new database fields, and re-ran
>> python manage.py makemigrations
to create a migrations file which adds the new fields to the database

Finally
>> python manage.py migrate
applied the changes to the database, adding the new fields

So far so good

Using PHPMyAdmin I can now see the new database fields in the MySQL database

However, I have yet to apply the changes to the live system. I’ll do this once the changes are made and tested. But first I’ll check the revision history on the server and I’ll backup the database, just in case

Django – a very simple unit test

Test Driven Development (TDD) is a methodology for developing programs where you first write the test, and then write just enough code to make your program pass all tests

TDD makes it far less likely that fixing one bug introduces a new one, or that a change reintroduces an old bug. It means that you can continuously test your code

Django has a built-in testing system, with two different approaches: doctests and unit tests. Here is how to create and run a very simple unit test

  1. Create a new app – python manage.py startapp testtest
  2. Edit …/testtest/tests.py, and add the following lines:
    class SimpleTest(TestCase):
    def test_basic_addition(self):
    self.failUnlessEqual(1 + 1, 2)
  3. python manage.py test
  4. This should show “OK”
  5. Change the last line to self.failUnlessEqual(1 + 1, 3)
  6. Re-run the test
  7. You should now get an error: AssertionError: 2 != 3

Using PostgreSQL with Django

Here is how go set up Django to work with PostgreSQL

  1. Install necessary libraries, etc
    1. sudo apt-get install libpq-dev
    2. sudo apt-get install python-dev
    3. sudo apt-get install postgresql-contrib
  2. Create a new database and user
    1. sudo su – postgres
    2. createdb djangodev
    3. createuser -P djangodev
      1. Enter password, twice
    4. psql
      1. postgres=#  GRANT ALL PRIVILEGES ON DATABASE djangodev TO djangodev;
      2. \q
  3. Make sure you have your virtual environment activated
  4. pip install psycopg2
  5. Open up the project’s settings.py and change the DATABASES to:
    DATABASES = {
    ‘default’: {
    ‘ENGINE’: ‘django.db.backends.postgresql_psycopg2’, # Add ‘postgresql_psycopg2’, ‘mysql’, ‘sqlite3’ or ‘oracle’.
    ‘NAME’: ‘djangodev’, # Or path to database file if using sqlite3.
    ‘USER’: ‘djangodev’,
    ‘PASSWORD’: ‘904ojioe_=3D’,
    ‘HOST’: ‘localhost’, # Empty for localhost through domain sockets or ‘127.0.0.1’ for localhost through TCP.
    ‘PORT’: ”, # Set to empty string for default.
    }
    }
  6. python manage.py syncdb

Installing PostgreSQL

PostgreSQL seems to be the most popular DBMS (database management system) with Django developers, although MySQL is also used a lot

To install PostgreSQL, I used Linux Mint’s Software Manager (search for “postgresql”)

I also installed pgAdmin III, “a database design and management application for use with PostgreSQL” using the same method

I used to work on a system which used PostgreSQL, but that is a long time ago, so I had to ask the Internet to remind me how to get it going. Here is how to get it started

  1.  Using the Software Manager, install PostgreSQL and pgadmin3
  2. Set the PostgreSQL password, for the postgres user:
    1. sudo -u postgres psql
    2. postgres=#      \password postgress
    3. (set the password)
    4. \q
  3. Start Programming -> pgAdmin III
    1. Click on “Server Groups”
    2. Click on the plug icon (top left hand corner)
    3. Name: Local DBMS (or whatever you want to call it)
    4. Host: localhost
    5. Port, Service, Maintenance DB: leave as is
    6. Username: postgres
    7. Password: the password you set in the step above
    8. Click on “Ok”

You should now be able to view your PostgreSQL server in pgAdmin, and use it to manage users, databases, etc

Serving static files in Django

From https://docs.djangoproject.com/en/dev/howto/static-files/: For a production site you probably want to configure the server to serve static files directly, without going through Django. For a test site, you can use the following approach

  1. In settings.py, url for requesting any static files is defined as follows:
    STATIC_URL = ‘/static/’

    1. This is the default, so you do not need to add it in, but you may want to change it
  2. To specify the location of the static files, add the following code to settings.py:
    STATICFILES_DIRS = (
    os.path.join(BASE_DIR, “static”),
    )

    1. Django will now look for static files in (project root)/static
  3. Create the static file, in my case in <dev root>/Projects/DjangoText/static/testimage.jpg
  4. In your html template file or generated html, before including a static file, add the following:
    {% load staticfiles %}
  5. Include the static file as follows:
    <img src=”{% static “testimage.jpg” %}” “/>

    1. Note: When Django sees the “static” marker, it knows to extend the path to include the location of the static folder
    2. Note the double double quotes
  6. Refresh the screen to test

 

Basic Django templates

Instead of using Python to create all the HTML, Django can load and render templates

  1. In settings.py specify the location of the templates, e.g.
    TEMPLATE_DIRS = [
    os.path.join(BASE_DIR, ‘templates’),
    ]
  2. Create the folder within the projects folder,
    i.e. <dev root>/Projects/DjangoTest/templates
  3. In the ../templates folder, create a new file, say testtemplate.html, and create a very basic html page, e.g.
    <html>
    <body>
    <p>Test template</p>
    </body>
    </html>
  4. Create a view to test the template, in the views.py
    from django.template.loader import get_template
    from django.template import Context

    def templateDemo(request):
    template = get_template(‘testtemplate.html’)
    html = template.render(Context({}))
    return HttpResponse(html)

  5. Set up the url, in the urls.py:
    url(r’^templatedemo/’, templateDemo),
  6. In the same file, import the view
    from DjangoTest.views import templateDemo
  7. Test it out – point your browser to http://localhost:8000/templatedemo/
    You should see a page with “Test template” and nothing else
  8. Now we’ll put some variable data in.
    1. In the template file (testtemplate.html), add the following line:
      <p>And the answer is: {{answer}}</p>
    2. Change views.py, line to:
      html = template.render(Context({‘answer’: 25}))
    3. Refresh the browser window. You should now see:
      Test template
      And the answer is: 25

Your second Django project

If you’ve followed the steps in the previous blog posts, you should have your first basic Django project up and running. You’ll also have various bits and pieces already installed, ready for your next projects. Here is a brief reminder of what to do for your next project:

  1. Choose a project name, e.g. SecondDjango
  2. Create a virtual environment, and use it
    1. cd <dev root>/envs
    2. virtualenv SecondDjango
    3. source SecondDjango/bin/activate
  3. Install Django and python-mysql into your new virtual environment
    1. (SecondDjango) pip install MySQL-python
    2. (SecondDjango) pip install Django
  4. Create your project
    1. (SecondDjango) cd <dev root>/Projects
    2. (SecondDjango) django-admin.py startproject SecondDjango
  5. Create a new database and user, and enter the details in the settings.py file
  6. Create a new app, add to the INSTALLED_APPS in settings
  7. Create your models, add them to the admin site in the app’s admin.py
  8. Create a superuser: python manage.py createsuperuser
  9. Start the test server (runserver) and check out the admin panel (http://localhost:8000/admin/)