Running Django on Windows (with performance tests)

Django is a trendy Python-based framework for web applications. Its ever-growing popularity owes to availability of quick development tools, inbuilt administrative interface and high operation speed. Unfortunately, until now there was no easy tried-and-true way to deploy and run Django-apps on IIS web-server.

Introducing Helicon Zoo – a repository of popular web-frameworks for Microsoft IIS. It bases on Microsoft Web Platform Installer (WebPI) technology to deploy apps. WebPI also handles different dependencies and takes care of installation of necessary components, such as Python, Django, various database drivers and modules. And, of course, Helicon Zoo itself which sticks it all to MS IIS 7.

How to use

Web development process implies use of two relatively independent environments – development and production. Helicon Zoo may be used in production as well as on developer’s machine, or in both places. In either case the sequence of actions might be:

To start, you need to download Web Platform Installer from Microsoft website (http://www.microsoft.com/web/downloads/platform.aspx) and install it. WebPI already includes wide range of frameworks and applications for IIS like PHP, ASP.NET, WordPress, Drupal, phpBB. To launch Helicon Zoo add new feed to WebPI:

Run WebPI and click Options

In «Display additional scenarios» box put http://www.helicontech.com/zoo/feed and click Add feed

Then go to Zoo -> Packages and install Python Hosting Package:

The installation process may take time and will install all required dependencies:

After installation is completed, create a new web site using IIS Manager. Then go to this web site folder using command line. Run the following command to create new Django application in the selected location:

C:\Python27\Scripts\django-admin.py startproject mysite

This command will create an empty Django application in the /mysite folder of the IIS web site. Now put the following web.config file into the root of the IIS web site (near /mysite folder and not inside it). Note “DJANGO_SETTINGS_MODULE” variable with the name of the application just created:

<?xml version="1.0" encoding="UTF-8"?> 
<configuration> 
 <system.webServer>
  <heliconZoo>
   <application name="django.project" >
    <environmentVariables>
     <add name="PYTHONPATH"
          value="%APPL_PHYSICAL_PATH%;%APPL_PHYSICAL_PATH%\python_modules\Lib\site-packages;%PYTHONPATH%" />
     <add name="DJANGO_SETTINGS_MODULE" value="mysite.settings" />
     <add name="django.root" value="%APPL_VIRTUAL_PATH%" />
     <add name="DEPLOY_FILE" value="deploy.py" />
     <add name="DEPLOY_LOG" value="log\deploy.log" />
     <add name="WATCH_FILE_CHANGES_MASK" value="*.py" />
    </environmentVariables>
   </application>
  </heliconZoo>
  <handlers>
   <add name="django.project#x86" scriptProcessor="python.2.7.wsgi"
        path="*" verb="*" modules="HeliconZoo_x86"
        preCondition="bitness32" resourceType="Unspecified" requireAccess="Script" />
   <add name="django.project#x64" scriptProcessor="python.2.7.wsgi"
        path="*" verb="*" modules="HeliconZoo_x64"
        preCondition="bitness64" resourceType="Unspecified" requireAccess="Script" />
  </handlers>
 </system.webServer>
</configuration>

Then navigate your browser to the created web site, you should see Django welcome page:

This screen symbolizes that Python, Django and the rest of things needed to run the apps were successfully installed on your PC. Now, if it’s a dev environment, you can start creating application. After you make some changes to the app, you may upload it to production server with all files, folders, necessarily web.config. The base requirement is that a server should have Python Hosting Package installed as described above.

Installing modules

Usually the application with Helicon Zoo is configured to load additional modules from the \python_modules\ directory inside web application folder. This is different from traditional development practice when users tend to install everything on the server globally. So if you install additional modules for your application don’t forget to install them locally into the \python_modules\ directory.  Here is a command line example, that command should be run from the web application root folder:

pip install --install-option="--prefix=python_modules" django==1.3

For convenience you can use requirements.txt file with the deploy.py file provided below instead of installing required modules manually. Simply copy the following deploy.py file into the root of your web application. Together with web.config provided above and requirements.txt file this will ensure all requirements for your application are installed in the local web application folder, providing no external dependencies except of Python Hosting Package. The deploy.py file will be executed every time the IIS application restarts and will also do database migrations if needed.

deploy.py example:

# This file is example of deploy.py
# The file is executed once on the first request after every restart of IIS application.
# The file output is redirected to log file described in DEPLOG_LOG environment variable.

import sys
import os
import os.path
import shutil
import django

PROJECT_DIR = os.path.dirname(__file__)
os.chdir(PROJECT_DIR)

PIP_EXE = os.path.join(os.path.dirname(sys.executable), 'scripts\\pip.exe')

# update APPDATA env for pip
os.environ['APPDATA'] = os.path.join(PROJECT_DIR, 'python_modules')

# install requirements to local folder
os.system('{0} install --install-option="--prefix={1}\python_modules" --requirement=requirements.txt'.format(PIP_EXE, PROJECT_DIR))

# run manage.py syncdb --noinput
os.system(sys.executable + " manage.py syncdb --noinput")

# run manage.py migrate
os.system(sys.executable + " manage.py migrate")

# copy admin media files
if not os.path.exists(os.path.join(PROJECT_DIR, 'static', 'admin', 'css', 'base.css')):
    print 'Copying admin media files'
    shutil.copytree(
      os.path.join(os.path.dirname(django.__file__), 'contrib', 'admin', 'media'),
      os.path.join(PROJECT_DIR, 'static', 'admin'))
else:
    print 'Admin media files already copied'

requirements.txt example:

This is only example of the file format. Please use your application specific requirements to fill this file.

django==1.3
south
tornado>2.0

Since WATCH_FILE_CHANGES_MASK option was specified in web.config, the Python application will be restarted every time you modify any *.py file inside web application folder. This is very helpful for both development and production environment. In production environment this will call deploy.py script, install application requirements into local folder and do database migrations every time you upload your changes to the server. Please note this could be a time consuming operation, especially when lot of modules need to be downloaded and updated, so it could take some time for the application to start when you first upload it to the production server.

Using WebMatrix

Another way to go is to use WebMatrix for development. Instead of creating empty Django project manually go to Zoo->Packages and install WebMatrix Templates and Python Hosting Package (if not previously installed).

After that run WebMatrix and create new application from template. You will be proposed number of templates to choose from. Please select “Django site” and create it.

This will create new Django application template, with directory structure, deploy.py file and all other files listed above and needed for comfortable development.

Then, again, to deploy your web site to a remote IIS server Python Hosting Package has to be installed on this server to run Django applications.

Under the hood

Helicon Zoo core is represented by native IIS module acting as a bridge between IIS web-server and frameworks using Ruby, Python, Perl etc. the module makes use of FastCGI protocol which is recognized as quick and accurate means for interaction between web-apps and web-server. This interaction is asynchronous and uses I/O Completion Port technology. For transport Zoo uses either named pipes or TCP-sockets. Web-servers supported include IIS 7, IIS 7.5 and IIS Express.

Basic Helicon Zoo configuration is stored in <heliconZooServer> section of applicationHost.config. This section includes descriptions of all FastCGI-drivers to be operated via Zoo. Here’s an example of driver description for running Python wsgi-apps (Django-apps in particular), zoofcgi.py is a worker written in Python that provides transport through named pipes:

The web.config file is used to enable and configure Helicon Zoo for particular application. Some important comments on the above config:

• PYTHONPATH — optional path to Python modules. In the example it points to application root, the python_modules\Lib\site-packages subfolder and also includes value defined by user.
• DJANGO_SETTINGS_MODULE — optional path to a file with Django application settings. By default it’s settings.py.
• django.root — virtual path of the application. In the example its value is taken from the %APPL_VIRTUAL_PATH% variable.
• DEPLOY_FILE — optional path to a a script which is run before application startup and every time IIS application pool recycles. The script usually contains deployment instructions to satisfy application dependencies and database migrations.
• DEPLOY_LOG — optional path to a text file containing the output of the deploy script. It’s recommended to set this variable if you use deploy script.
• WATCH_FILE_CHANGES_MASK — files mask to watch for chnages. Worker is restarted every time a file of specified type changes. In the example all .py files are monitored.
• scriptProcess="python.2.7.pipe" is a reference to FastCGI-drived defined in applicationHost.config;

Static content

Django framework is not suitable for fast and safe processing of static files (images, scripts, css). Statics must be processed directly by web-server. To do so all static files must be stacked in one folder and Django request handler for this folder is then switched off like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <!-- 
      This removes Helicon Zoo handler and makes IIS processing static files.
      -->
      <remove name="django.project#x64" />
      <remove name="django.project#x86" />
    </handlers>
  </system.webServer>
</configuration>

Sample Django-project for IIS 7

Let’s have a Django-project consisting of 3 apps:

• site — the site itself with templates, statics, urls.py nad settings.py
• blog
• store

Static files located in site/media are available in the /media/ virtual folder (i.e. MEDIA_URL=’/media/’)

Having installed Blank Django Project in the root of the site and having copied our django-project, we’ll get the following structure:

/media/ is a virtual folder pointing at site/media. Zoo module is switched off for this folder as mentioned above. PYTHONPATH in root web.config points at site root and DJANGO_SETTINGS_MODULE is set to ‘site.settings’:

...
  <heliconZoo>
    <application name="django.project" >
     <environmentVariables>
      <add name="PYTHONPATH" value="%APPL_PHYSICAL_PATH%" />
      <add name="DJANGO_SETTINGS_MODULE" value="site.settings" />
     </environmentVariables>
    </application>
   </heliconZoo>
...

Performance

And now performance test for dessert. Testing machine acting as server was Core 2 Quad 2.4 GHz, 8 Gb RAM, Gigabit LAN. To generate load we used more powerful PC with Apache Benchmark. To measure Apache and Nginx performance Ubunthu 11.04 Server x64 was used. IIS 7 tests ran on Windows Server 2008 R2. No virtual machines – only bare hardware. As transport on Nginx we used the most advanced uwsgi, as well as wsgi and fast_cgi for comparison. On IIS 7 we’ve also compared with PyISAPIе.

There where two Django scripts created as testing pages. The first one outputs current time in high resolution; this is done to ensure pages are not taken from cache. The second one does the same but previously saves the result into database. It’s all done using templates in order to apply real Django infrastructure; DB used is MySQL. All settings were left default as the task was to test the most common configurations. Here are results (in requests per second):

No surprises here as Python performance on Windows may be slower than Ubunthu version. Taking this into consideration Helicon Zoo transport performance should be really high. Uwsgi is ahead probably due to closer integration with Django.

The results for the second script are not that smooth. Why Nginx + fcgi + MySQL showed only 175 requests per second remains unknown. MySQL on Windows score is also frustrating, although on shared hosting the problem might not be that critical. The thing is that performance drops due to internal MySQL locks, while the server is not even loaded for 20% while generating these 104 requests per seocnd. It’s reasonable to assume that by increasing number of sites on the server and consequently the number of DBs, if they do not interlock with each other, the total server performance will be acceptable.

Thus we decided to add MS SQL Express into the tests. The result was easy to explain with Python and its database driver being the bottleneck, though in general the picture is quite promising. Unfortunately PyISAPIe was unable to work with MS SQL Express and was excluded from tests.

It is worth to mention the ability of IIS 7 to handle great number of connections. IIS 7 + Helicon Zoo easily held thousands of concurrent connections, we simply didn’t have testing powers to generate enough connections to trigger any problems in this test. Ubuntu with default settings started throwing connection failures when the number of connections increased. Moreover, Apache appeared to be greedy for memory. During the test with the number of connections going up Apache swallowed about 3 GB in 20 seconds.

Resume

Suggested solution proved stable and efficient. It will perfectly fit development environment as well as production. Advantageous is possibility of using Helicon Zoo by different Windows shared hosting providers to offer Django services to their clients. Hope is that with the growth number of Django servers on Windows Python developers will pay more attention to code debugging and optimization for Windows platform. And the army of existing Windows-developers can also contribute to current open-source projects.