Bryan eb06dd4d68 moving some docs into README and others under docs/, removing redundant or out of date docs for #179. | 10 years ago | |
---|---|---|
bin | 11 years ago | |
confs | 10 years ago | |
docs | 10 years ago | |
karmaworld | 10 years ago | |
reqs | 10 years ago | |
var | 11 years ago | |
.gitignore | 10 years ago | |
README.md | 10 years ago | |
Vagrantfile | 10 years ago | |
fabfile.py | 10 years ago | |
manage.py | 10 years ago |
Description: A django application for sharing and uploading class notes.
Copyright: FinalsClub, a 501c3 non-profit organization
License: GPLv3 except where otherwise noted
Contact: info@karmanotes.org
v3.0 of the karmanotes.org website from the FinalsClub Foundation
KarmaNotes is an online database of college lecture notes. KarmaNotes empowers college students to participate in the free exchange of knowledge.
Before doing anything, you'll need the code. Grab it from github.
Clone the project from the central repo using your github account:
git clone git@github.com:FinalsClub/karmaworld.git
If you aren't using a system setup for github, then grab the project with this command instead:
git clone https://github.com/FinalsClub/karmaworld.git
Generally speaking, this will create a subdirectory called karmaworld
under
the directory where the git
command was run. This git repository directory
will be referred to herein as {project_root}
.
There might be some confusion as the git repository's directory will likely be
called karmaworld
(this is {project_root}
), but there is also a karmaworld
directory underneath that ({project_root}/karmaworld
) alongside files like
fabfile.py
({project_root}/fabfile.py
) and README.md
({project_root}/README.md
).
Notice: This software makes use of external third party services which require accounts to access the service APIs. Without these third parties available, this software may require considerable overhaul.
This software uses Filepicker.io for uploading files. This requires an account with Filepicker.
Filepicker requires an additional third party file hosting site where it may send uploaded files. For this project, we have used Amazon S3.
Filepicker will provide an API key. This is needed by the software.
This software uses Amazon S3 as a third party file hosting site. The primary use case is a destination for Filepicker files. The software won't directly need any S3 information for this use case; it will be provided directly to Filepicker.
A secondary use case for S3 is hosting static files. The software will need to update static files on the S3 bucket. In this case, the software will need the S3 bucket name, access key, and secret key.
The code assumes S3 is used for static files in a production environment. To obviate the need for hosting static files through S3 (noting that it still might be necessary for Filepicker), a workaround was explained in this Github ticket.
That workaround is repeated here. Make the following changes to
{project_root}/karmaworld/settings/prod.py
:
INSTALLED_APPS
STATIC_URL
to '/assets/'
INSTALLED_APPS
and STATIC_URL
)add this to the nginx config:
location /assets/ {
root /var/www/karmaworld/karmaworld/;
}
This software uses Google Drive to convert documents to and from various file formats.
A Google Drive service account with access to the Google Drive is required. Thismay be done with a Google Apps account with administrative privileges, or ask your business sysadmin.
These are the instructions to create a Google Drive service account: https://developers.google.com/drive/delegation
When completed, you'll have a file called client_secrets.json
and a p12 file
which is the key to access the service account. Both are needed by the software.
Twitter is used to post updates about new courses. Access to the Twitter API will be required for this task.
If this Twitter feature is desired, the consumer key and secret as well as the access token key and secret are needed by the software.
If the required files are not found, then no errors will occur.
If you need to setup the project for development, it is highly recommend that you grab create a development virtual machine or (if available) grab one that has already been created for your site.
The host machine is the system which runs e.g. VirtualBox, while the virtual machine refers to the system running inside e.g. VirtualBox.
Create a virtual machine with your favorite VM software. Configure the virtual machine for production with the steps shown in the Production Install section.
Vagrant supports a variety of virtual machine software and there is additional support for Vagrant to deploy to a wider variety. However, for these instructions, it is assumed Vagrant will be deployed to VirtualBox.
Configure external dependencies on the host machine:
{project_root}/karmaworld/secret/
:
cp filepicker.py.example filepicker.py
)db_settings.py
(Vagrant takes care of that one)drive.p12
(this filename and location may be changed in drive.py
)*.py
in secret/
are never added to the git repo.
(.gitignore should help warn against taking this action)Install VirtualBox
Install vagrant 1.3 or higher
Use Vagrant to create the virtual machine.
cd {project_root}
, type vagrant up
Connect to the virtual machine with vagrant ssh
Note:
Port 80 of the virtual machine will be configured as port 6659 on the host
system. While on the host system, fire up your favorite browser and point it at
http://localhost:6659/
. This connects to your host system on port 6659, which
forwards to your virtual machine's web site.
On the virtual machine, type cd karmanotes
to get into the code repository.
In the code repo of the VM, type fab -H 127.0.0.1 first_deploy
During this process, you will be queried to create a Django site admin. Provide information. You will be asked to remove duplicate schools. Respond with yes.
These steps are taken care of by automatic utilities. Vagrant performs the first subsection of these instructions and Fabric performs the second subsection. These instructions are detailed here for good measure, but should not generally be needed.
git
7zip
(for unzipping US Department of Education files)PostgreSQL
(server and client)nginx
libxslt
and libxml2
(used by some Python libraries)RabbitMQ
(server)memcached
Python
PIP
virtualenv
virtualenvwrapper
(might not be needed anymore)On a Debian system supporting Apt, this can be done with:
sudo apt-get install python-pip postgresql python-virtualenv \
virtualenvwrapper git nginx p7zip-full \
postgresql-server-dev-9.1 libxslt1-dev libxml2-dev \
libmemcached-dev python-dev rabbitmq-server
Generate a PostgreSQL database and a role with read/write permissions.
Modify configuration files.
{project_root}/karmaworld/settings/prod.py
{project_root}/karmaworld/secret/
.
cp filepicker.py.example filepicker.py
)PROD_DB_USERNAME
, PROD_DB_PASSWORD
, and PROD_DB_NAME
inside db_settings.py
match the role, password, and database
generated in the previous step.drive.p12
(this filename and location may be changed in drive.py
)*.py
in secret/
are never added to the git repo.
(.gitignore should help warn against taking this action)Make sure that /var/www exists, is owned by the www-data group, and that the desired user is a member of the www-data group.
Configure nginx with a proxy_pass
to port 8000 (or whatever port gunicorn
will be running the site on) and any virtual hosting that is desired.
Here is an example server file to put into /etc/nginx/sites-available/
server {
listen 80;
# don't do virtual hosting, handle all requests regardless of header
server_name "";
client_max_body_size 20M;
location / {
# pass traffic through to gunicorn
proxy_pass http://127.0.0.1:8000;
}
}
Configure the system to start supervisor on boot. An init script for
supervisor is in the repo at {project_root}/karmaworld/confs/supervisor
.
update-rc.d supervisor defaults
is the Debian command to load the init
script into the correct directories.
Make sure {project_root)/var/log
and {project_root}/var/run
exist and
may be written to, or else put the desired logging and run file paths into
{project_root}/confs/prod/supervisord.conf
Create a virtualenv under /var/www/karmaworld/venv
Change into the virtualenv with . /var/www/karmaworld/venv/bin/activate
.
Within the virtualenv:
Update the Python depenencies with pip -i {project_root}/reqs/prod.txt
Setup the database with python {project_root}/manage.py syncdb --migrate
Collect static resources and put them in the static hosting location with
python {project_root}/manage.py collect_static
The database needs to be populated with schools. A list of accredited schools may be found on the US Department of Education website: http://ope.ed.gov/accreditation/GetDownloadFile.aspx
Alternatively, use the built-in scripts while in the virtualenv:
Fetch USDE schools with
python {project_root}/manage.py fetch_usde_csv ./schools.csv
Upload the schools into the database with
python {project_root}/manage.py import_usde _csv ./schools.csv
Clean up redundant information with
python {project_root}/manage.py sanitize_usde_schools
Startup supervisor
, which will run celery
and gunicorn
. This may be
done from within the virtualenv by typing
python {project_root}/manage.py start_supervisord
If everything went well, gunicorn should be running the website on port 8000 and nginx should be serving gunicorn on port 80.
If you have installed a virtual machine using vagrant up
, you can connect
to it by running vagrant ssh
from {project_root}
.
To access the website running on the VM, point your browser at http://localhost:6659/ using your host computer.
Port 6659 on your local machine is set to forward to the VM's port 80.
Fun fact: 6659 was chosen because of OM (sanskrit) and KW (KarmaWorld) on a phone: 66 59.
Once connected to the virtual machine by SSH, you will see karmaworld
in
the home directory. That is the {project_root}
in the virtual machine.
cd karmaworld
and then use git fetch; git merge
and/or git pull origin
as
desired.
The virtual machine's code repository is set to use your host machine's local repository as the origin. So if you make changes locally and commit them, without pushing them anywhere, your VM can pull those changes in for testing.
This may seem like duplication. It is. The duplication allows your host machine to maintain git credentials and manage repository access control so that your virtual machine doesn't need sensitive information. Your virtual machine simply pulls from the local repository on your local file system without needing credentials, etc.
Please see vagrant documentation for more information on how to use the vagrant CLI to manage your development VM.
We have setup Django to use
south for migrations. When
changing models, it is important to run
python {project_root}/manage.py schemamigration
which will create a migration
to reflect the model changes into the database. These changes can be pulled
into the database with python {project_root}/manage.py migrate
.
Sometimes the database already has a migration performed on it, but that
information wasn't told to south. There are subtleties to the process which
require looking at the south docs. As a tip, start by looking at the --fake
flag.
A number of assets have been added to the repository which come from external
sources. It would be difficult to keep a complete list in this README and keep
it up to date. Software which originally came from outside parties can
generally be found in {project_root}/karmaworld/assets
.
Additionally, all third party Python projects (downloaded and installed with pip) are listed in these files:
{project_root}/reqs/common.txt
{project_root}/reqs/dev.txt
{project_root}/reqs/prod.txt
KarmaNotes.org is a project of the FinalsClub Foundation with generous funding from the William and Flora Hewlett Foundation
Also thanks to rdegges for the django-skel template