Councilmatic Starter Template

This repo provides starter code and documentation for new Councilmatic instances.

About Councilmatic

The councilmatic family is a set of web apps for keeping tabs on city representatives and their legislative activity.

Councilmatic started as a Code for America project by Mjumbe Poe, who designed the earliest version of a Councilmatic site – Councilmatic Philadelphia. DataMade then implemented Councilmatic in New York City (in partnership with the Participatory Politics Foundation), Chicago (in partnership with the Sunlight foundation), and Los Angeles (in partnership with the Los Angeles County Metropolitan Transportation Authority).

To simplify redeployment of Councilmatic instances, DataMade identified two necessities: (1) Open Civic Data, a data standard for describing people, organizations, events, and bills, and (2) the abstraciton of core functionality into a separate app. And so, DataMade built django-councilmatic – a django app with base functionality, common across all cities, which implements the OCD data standard.

With this template, you can create a Councilmatic site for your own city, built using the django-councilmatic app. Read on!

Getting started

1. Install OS Level dependencies

• Python 3.4+
• PostgreSQL 9.4+

2. Clone this project

git clone https://github.com/datamade/councilmatic-starter-template.git yourcity_councilmatic

3. Make a virtualenv and install dependencies

We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.

Once you have virtualenvwrapper set up, run the following commands in your terminal:

cd yourcity_councilmatic
mkvirtualenv councilmatic
pip install -r requirements.txt

Afterwards, whenever you want to use this virtual environment, run:

workon yourcity_councilmatic

4. Set up your database

Initialization

Before you can run the website, you need to create a database and install PostGIS.

createdb yourcity_councilmatic
psql -d yourcity_councilmatic -c "CREATE EXTENSION postgis"

Then, run migrations. (Be sure that you are "working on" the correct virtual environment.)

python manage.py migrate --no-initial-data

Finally, create an admin user. Set a username and password when prompted.

python manage.py createsuperuser

Find legislative data

django-councilmatic leverages, and in some instances, lightly extends the Open Civic Data standard, implemented in Django as python-opencivicdata. The next step toward running a new Councilmatic instance is to locate data about your city and use it to populate these OCD models.

How you do that is up to you. At DataMade, we maintain a series of external municipal scrapers that retrieve data from Legistar-backed sites.

If your city runs a Legistar-backed site, see scrapers-us-municipal for several examples of scrapers that will populate a given database with legislative data in the format required by Councilmatic. These scrapers leverage python-legistar-scraper to scrape Legistar and the pupa framework to shape and import data.

If your city does not run a Legistar-backed site, we still recommend using pupa as a legislative scraping framework. You can add your scrapers to your Councilmatic repository, or compose them elsewhere.

Next, install and create a settings file for pupa –

pip install pupa
touch pupa_settings.py

– and add the following settings:

# Leave these blank
OCD_CITY_COUNCIL_NAME = ''
CITY_COUNCIL_NAME = ''
STATIC_PATH = ''

INSTALLED_APPS = (
    'django.contrib.contenttypes',
    'opencivicdata.core.apps.BaseConfig',
    'opencivicdata.legislative.apps.BaseConfig',
    'pupa',
    'councilmatic_core'
)

# Change this if you called your database something different
DATABASE_URL = 'postgres:///yourcity_councilmatic'

Finally, initialize pupa.

pupa dbinit us
pupa init YOUR_CITY_SCRAPER

Once you've filled out the prompts, you will see a new directory called YOUR_CITY_SCRAPER. Inside, you'll find the scaffolding for the necessary scrapers to populate the database.

Define the scrape methods in each of the resulting scraper classes, then run pupa update YOUR_CITY to import data. If you aren't sure where to start, the City Scrapers project includes many examples of scraping all sorts of government websites.

N.b., if you wish to make any changes to the models, see the django-councilmatic README for information on approaches.

Add metadata

django-councilmatic comes with management commands to add headshots and geography shapes to Person objects and Post objects, respectively.

If you've imported people with headshot URLs, run update_headshots to download the images and store them where Django knows where to find them.

python manage.py update_headshots

Likewise, if you've imported Post objects and you'd like to include geographic boundaries that pertain to them, create a GeoJSON file containing shapes for each division represented in your instance, then run the import_shapes command. See the GeoJSON file in chi-councilmatic for an example of the expected format.

python manage.py import_shapes /path/to/your/shapes.geojson

5. Rename the "city" app

Now that you have the data, set up your instance.

Inside the git repository that you cloned above, you should see a folder called city. Rename this folder to something that makes sense for your project, e.g., "chicago."

mv city yourcity

Now, update the main settings file - councilmatic/settings.py. First, in INSTALLED_APPS, add the name of the folder that you just renamed:

INSTALLED_APPS = (
    ...
    ~'city',~
    'yourcity',
)

Then, change the TIME_ZONE. You can use this list to find the correct formatting.

TIME_ZONE = 'America/Chicago'

6. Update city-specific settings

Look for councilmatic/settings_jurisdiction.py. This settings file tells your Councilmatic instance how to populate different parts of the UI and get fresh data from the OCD database. The following table explains why and how to adjust these values.

7. Update deployment settings

Look for councilmatic/settings_deployment.py.example. Make a copy of it so that you can customize it for your city:

cp councilmatic/settings_deployment.py.example councilmatic/settings_deployment.py

This file is important! It's where you keep the parts of your Councilmatic that should not end up in version control (e.g., passwords, database connection strings, etc., which will vary based on where the app is running and which you probably won't want other people to know). Most of these variables are generic Django settings, which you can look up in the Django docs; but some of these variables require customization. This table shows only the variables that you need to customize.

8. Add a favicon

Get an image (suggested 310x310), and transform it in device-specific favicon images using this site. Then, move the images into <city_app>/static/images/icons.

9. Run Councilmatic locally

Now, you are ready to run your unique version of Councilmatic!

python manage.py runserver

Navigate to http://localhost:8000/.

Set up search with Haystack and Solr

On a Councilmatic site, users can search bills according to given query parameters. To power our searches, we use Django Haystack to connect with Solr, an open source tool, written in Java.

Run Solr

Because it's a bit of a heavy dependency, we recommend running a containerized instance of Solr. See the Solr image in DockerHub for more information and startup instructions. Don't forget to update your Haystack config in councilmatic/settings_deployment.py, if you map a port other than the Solr default (8983) to your container!

If you prefer to run Solr locally, see the Solr documentation for startup instructions.

Haystack commands to know

Haystack provides several management commands that make it easy to change and add data to your search index.

To make your data searchable:

python manage.py rebuild_index

If, during the course of development, you need to make changes to the fields that are indexed in yourcity/search_indexes.py, you need to regenerate the Solr schema:

python manage.py build_solr_schema > solr_scripts/schema.xml

Remove and recreate your Solr container to capture changes in the Solr schema.

Errors/Bugs

If something is not behaving intuitively, it is a bug, and should be reported. Report it here.

Note on Patches/Pull Requests

We welcome your ideas and feedback! Here's how to make a contribution:

• Fork the project.
• Make your feature addition or bug fix.
• Commit - please be descriptive, but concise.
• Send a pull request. Bonus points for well-titled topic branches!

Copyright

Copyright (c) 2019 Participatory Politics Foundation and DataMade. Released under the MIT License.