分类 web开发 下的文章

本站目前的开发环境是 Python3.5.2+Django1.10.6+Sqlite3+Centos7.5+Nginx1.12.2+Gunicorn

blog虽然已经上线测试有一段时间了(2018.8.7上线),但是项目部署的笔记由于内容较多,没时间整理。今天抽空整理了一下午,发布出来供需要的同学借鉴参考。文中如有错误请多多指正

部署前的准备

服务器

要部署项目就要先购买服务器,如腾讯云,阿里云等等

注意:服务器在国内的话,必须要备案!不然无法使用。

阿里云优惠券:点击前往领取
腾讯云代金券:点击前往领取
腾讯云首页:点击直达
腾讯云西南区云产品促销链接:点击直达
腾讯云服务器购买链接:点击直达

域名

域名购买选择,可以到百度云,腾讯云,阿里云,GoDaddy,namecheap等等

购买域名后就是解析域名,打开域名解析面板,添加解析记录
记录类型:A
主机记录:test.markhoo.com,这个markhoo.com是你的一级域名,二级域名你可以设置为www.markhoo.com
解析线路:可以默认也可以选别的
记录值:填写你的服务器IP地址
TTL:域名生效时间,也就是多久生效
设置好以上内容,点击确定。解析完成

主机记录中这个markhoo.com是你的一级域名,二级域名你也可以设置为 www.markhoo.com 或者别的 xxx.markhoo.com 也可以。
注意:域名是否在国内购买不重要,只要服务器不在大陆购买,即使未备案照样可以使用。要是要解析的服务器是大陆服务商则必须要备案。备案后一个域名可以同时接入多个服务商。

域名解析记录类型说明:

  • A:将域名指向一个IPv4地址
  • AAAA:将域名指向一个IPv6地址
  • CNAME:将域名指向另外一个域名
  • NS:将子域名指定其他DNS服务器解析
  • MX:将域名指向邮件服务器地址
  • SRV:记录提供特定的服务的服务器
  • TXT:文本长度限制512,通常做SPF记录(反垃圾邮件)
  • CAA:CA证书颁发机构授权校验
  • 显性URL:将域名302重定向到另外一个地址
  • 隐性URL:与显性URL类似,但是会隐藏真实目标地址

xshell下载与安装

主要用于远程登录服务器进行相关操作。在官网注册后发送下载链接到你的注册邮箱,或者到搜索引擎搜索下载。
个人版免费,企业版收费。
下载后一路点击下一步,完成。

登录服务器步骤:
点击左上角'文件'--'新建'
然后弹出对话框
常规:名称,这个你可以随便取什么名字
协议:SSH,一般默认
主机:空,填上你的服务器IP地址
端口号:22,默认的就行
填好以上选项内容,点击左侧的'用户身份验证',然后看右侧,方法选择'Password',在下面输入你的服务器登录用户名和密码,最后点击下面的'连接'。
登录成功!进入界面。

百度网盘官网软件下载链接:
链接: https://pan.baidu.com/s/19K5q1JkmD9qnYaGRTALyLg 密码: r6hx

安装Nginx

首先用xshell登录服务器
通过 yum 安装 Nginx

yum install nginx

启动 Nginx 服务

systemctl start nginx

或者

systemctl start nginx.service
systemctl enable nginx.service

访问服务器IP地址,可以看到 nginx 的欢迎界面

Nginx相关命令:
停止nginx

nginx -s stop

重启nginx

nginx -s reload

Centos7.5安装Python3

由于centos7.5原本默认安装了Python2,而且这个Python2不能被删除,因为有很多系统命令,比如yum都要用到。
以下操作首先要用xshell登录到服务器
输入Python命令,查看可以得知是Python2.7.5版本

输入

which python

可以查看位置,一般是位于/usr/bin/python目录下。

下面介绍安装Python3的方法

首先安装依赖包

yum -y groupinstall "Development tools"
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

然后根据自己需求下载不同版本的Python3,我下载的是Python3.5.2

注:如果速度不够快,可以直接去官网下载,利用WinSCP等软件传到服务器上指定位置。

我的存放目录是/usr/local/python3,使用命令:

mkdir /usr/local/python3

建立好一个空文件夹,进入该目录

cd /usr/local/python3

使用一下命令下载压缩包

wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tar.xz

然后解压压缩包安装Python3

tar -xvJf Python-3.5.2.tar.xz
cd Python-3.5.2
./configure --prefix=/usr/local/python3
make && make install

最后创建软链接

ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

在命令行中输入python3测试

升级pip3命令:

pip3 install --upgrade pip

部署前的项目配置

Django 项目中会有一些 CSS、JavaScript 等静态文件,为了能够方便地让 Nginx 处理这些静态文件的请求,我们把项目中的全部静态文件收集到一个统一的目录下,这个目录通常位于 Django 项目的根目录,并且命名为 static。为了完成这些任务,需要在项目的配置文件里做一些必要的配置:

myblog/settings.py

# 其他配置...

STATIC_URL = '/static/'
# 加入下面的配置
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

STATIC_ROOT 指明了静态文件的收集目录,即项目根目录(BASE_DIR)下的 static 文件夹。

为了安全起见,在生产环境下需要关闭 DEBUG 选项以及设置允许访问的域名。打开 settings.py 文件,找到 DEBUG 和 ALLOWED_HOSTS 这两个选项,将它们设置成如下的值:

myblog/settings.py

DEBUG = False
ALLOWED_HOSTS = ['127.0.0.1', 'localhost ', '.markhoo.com']

ALLOWED_HOSTS 是允许访问的域名列表,127.0.0.1 和 localhost 是本地访问的域名,.markhoo.com 是访问服务器的域名(换成你自己的域名)。域名前加一个点表示允许访问该域名下的子域名,比如 www.markhoo.com test.markhoo.com 等二级域名同样允许访问。如果不加前面的点则只允许访问 markhoo.com。

项目还会依赖一些第三方 Python 库,为了方便在服务器上一次性安装,我们将全部依赖写入一个叫 requirements.txt 的文本文件中。激活本地的虚拟环境(如果你使用了虚拟环境的话),并进入项目的根目录,运行 pip freeze > requirements.txt 命令:

(myblog_env) D:Djangoprojectsmyblog>
pip freeze > requirements.txt

这时项目根目录下会生成了一个 requirements.txt 的文本文件,其内容记录了项目的全部依赖。由于有一些依赖包也有先后依赖关系,该调整的需要进去txt文件调整一下。

如果上传到服务器后修改,可以使用 'vi (文件名称)' 打开文件,然后点击键盘 'i' 键切换到输入模式。
修改完需要退出,先按键盘 'Esc' 键,然后输入 ':wq' 保存退出。如果不想修改又无法退出可以用 ':q!' 强制退出,这样不会保存你修改的内容
更多的vi、vim命令使用方法可以自行到搜索引擎搜索学习,这里就不一一讲解了。

将代码上传到 GitHub 、 Gitee 、 或者Coding

将代码上传到 GitHub 等代码托管平台,这样我们就可以方便地把代码拉取到服务器了。
如果访问慢或者打不开GitHub的话,可以上传到国内的Gitee或者Coding

Git的安装和上传代码的简单方法

安装Git可以到官网 https://git-scm.com 下载,安装的话可以一直下一步,完成。

首先你需要一个GitHub 、 Gitee 、 或者Coding账号,所以还没有的话先去注册吧!

https://github.com
https://gitee.com
https://coding.net

1.进入Github首页,点击New repository新建一个项目

2.填写相应信息后点击create即可

Repository name: 仓库名称

Description(可选): 仓库描述介绍

Public, Private : 仓库权限(公开共享,私有或指定合作者)

Initialize this repository with a README: 添加一个README.md

gitignore: 不需要进行版本管理的仓库类型,对应生成文件.gitignore

license: 证书类型,对应生成文件LICENSE

一般可以填写项目仓库名称,点击create就可以了。因为免费用户只有 Gitee 才可以创建私有仓库

3.创建好项目仓库后,在项目仓库页面点击Clone or dowload会出现一个地址,copy这个地址备用。

4.接下来就到本地操作了,首先右键你的项目,如果你之前安装git成功的话,右键会出现两个新选项,分别为Git Gui Here,Git Bash Here,这里我们选择Git Bash Here进入界面,我的本地项目名为myblog。

5.接下来输入如下代码(关键步骤),把github上面的仓库克隆到本地

git clone (粘贴上你之前复制的地址)

6.操作步骤5以后你的本地项目文件夹下面就会多出个文件夹,该文件夹名即为你github上面的项目名,如我多出了个myblogsite文件夹,我们把本地项目文件夹下的所有文件(除了新多出的那个文件夹不用),其余都复制到那个新多出的文件夹下。

7.接着继续输入命令 cd myblogsite,进入myblogsite文件夹

8.接下来依次输入以下代码即可完成其他剩余操作:

git add . (注:别忘记后面的.,此操作是把myblogsite文件夹下面的文件都添加进来)
git commit -m "提交信息" (注:“提交信息”里面换成你需要,如“first commit”)
git push -u origin master (注:此操作目的是把本地仓库push到github上面,此步骤需要你输入GitHub帐号和密码)

注意:数据库文件不要上传!要是SQLLITE3里面有你测试的重要数据,将可能会泄露。

设置服务器目录结构

接下来需要把代码上传到服务器了。我服务器上存放代码的目录结构一般是这样的:

/home/markhoo/
    sites/
        markhoo.com/
            myblog_env/
            myblog/

一台服务器可能部署多个网站,所有网站代码都放在 sites/ 目录下。markhoo.com/ 这个文件夹以网站的域名命名,便于区分。myblog_env 是 python 虚拟环境目录。myblog 是 Django 博客项目目录。
创建这个目录结构,注意目录名替换为你自己的域名,以后涉及到 markhoo.com 的地方通常都要替换你自己的域名,后面就不再一一指出了。
备注:上面的我的目录结构.可能会有新手看不懂,解释一下,就像下面这样一直创建下去

/home/markhoo/sites/markhoo.com/

最后的 myblog_env 和 myblog 是同级目录位于markhoo.com下,这里自己先不要创建。

创建目录的命令为: mkdir (文件夹名称)
删除目录的命令为: rmdir (文件夹名称)

安装virtualenv

为了不影响外界环境的清洁,所以我们使用虚拟环境来配置 Django 项目
首先用xshell登录到你的服务器,然后按一下步骤操作

pip3 install virtualenv

接下来创建虚拟环境,先进入到 markhoo.com 目录下,然后运行 virtualenv 命令创建虚拟环境:

[root@instance-pkb22o0h]# cd /home/markhoo/sites/markhoo.com
[root@instance-pkb22o0h markhoo.com]# virtualenv -p /usr/bin/python3 –-no-site-packages myblog_env

命令说明:
-p: 指定你要虚拟的Python版本,这里选择了本地的python3
–-no-site-packages:表示在建立虚拟环境时不将原版本中的第三方库拷贝过来,这样就能获得一个纯净的Python环境。
myblog_env:表明在该目录下,建立一个叫做myblog_env的虚拟环境,这样的命名方式,让你一眼就能看出这个虚拟环境是为谁建立的。

virtualenv的命令使用方法:
创建虚拟环境: $ virtualenv -p /usr/bin/python3.5 –-no-site-packages myblog_env
启动虚拟环境: $ source myblog_env/bin/activate 
退出虚拟环境: $ deactivate 
删除虚拟环境: $ rm –r myblog_env

检查一下虚拟环境是否创建成功,运行 ls 命令列出当前目录下的文件和文件夹,看到 myblog_env 这个文件夹说明虚拟环境创建成功。

[root@instance-pkb22o0h markhoo.com]# ls
myblog_env

venv的使用

由于我只是测试项目,所以使用了Python自带的venv。
先进入到 markhoo.com 目录下,然后运行 python3 -m venv myblog_env 创建虚拟环境:

[root@instance-pkb22o0h]# cd /home/markhoo/sites/markhoo.com
[root@instance-pkb22o0h markhoo.com]# python3 -m venv myblog_env

检查一下虚拟环境是否创建成功,运行 ls 命令列出当前目录下的文件和文件夹,看到 myblog_env 这个文件夹说明虚拟环境创建成功。

[root@instance-pkb22o0h markhoo.com]# ls
myblog_env

在虚拟环境中安装 django 以及依赖包并初始化项目

接着再从代码仓库把项目代码拉取过来,把 git clone 后的地址换成你自己的 GitHub 仓库地址!

[root@instance-pkb22o0h markhoo.com]# git clone https://github.com/MarkHoo/myblog.git

运行 ls 命令检查一下是否拉取成功:

[root@instance-pkb22o0h markhoo.com]# ls
myblog myblog_env

多了 myblog 文件夹(文件夹名称由你的 GitHub 仓库名决定),说明拉取成功了。

接下来执行命令,启动虚拟环境

[root@instance-pkb22o0h markhoo.com]# source myblog_env/bin/activate
(myblog_env) [root@instance-pkb22o0h markhoo.com]#

安装项目环境依赖

(myblog_env) [root@instance-pkb22o0h markhoo.com]# cd myblog
(myblog_env) [root@instance-pkb22o0h myblog]# pip3 install -r requirements.txt

收集静态文件
运行 python manage.py collectstatic 命令收集静态文件到 static 目录下

(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py collectstatic

生成数据库

(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py migrate

创建超级用户

(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py createsuperuser

然后就会出现以下内容,第一个是用户名,第二个是邮箱,第三和第四是密码(输入时没有任何显示,实际上已经输入了)。最后显示创建成功!

Username (leave blank to use 'root'): admin
Email address: admin@markhoo.com
Password:
Password (again):
Superuser created successfully.

备注:这里使用的是Django默认生成的sqlite3数据库,我使用了这个,简单写写博客够用了。

启动 Django

(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py runserver 0.0.0.0:8000

如果没有报错,就说明 Django 已经安装成功了,打开浏览器输入你的服务器IP:8000,就可以看到你的网站了

退出 Django

按 ctrl+c 退出 Django 服务

安装 Mysql (使用MySQL数据库同学可以借鉴)

安装并启动 mariadb

因为 CentOS 7 之后的版本都不在提供 Mysql 安装源,这里我们使用 mariadb 代替 mysql,依次执行下列命令

yum install mariadb mariadb-server -y
yum install MySQL-python -y
systemctl start mariadb

对 mariadb 进行初始化设置

  • 执行下面命令,根据提示操作
  • 设置新密码为 test
  • 默认密码为空,直接回车即可

mysql_secure_installation

使用设置的密码登陆 mariadb

  • 登陆 db,这里假设密码被设置为 admin123

mysql -uroot -ptest

创建一个数据库

create database myblog;

成功后,输入 exit 命令退出 db

exit

修改配置文件,与 Mysql 数据库相关联
编辑 myblog/settings.py

# Database
# https://docs.djangoproject.com/en/1.11/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'myblog',
        'PASSWORD':'admin123',
        'USER': 'root',
        'HOST':'127.0.0.1',
        'PORT':'3306',
    }
}

创建 Django 数据库

(myblog_env) [root@instance-pkb22o0h myblog]# cd /home/markhoo/sites/markhoo.com/myblog
(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py migrate

创建超级用户

(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py createsuperuser

然后就会出现以下内容,第一个是用户名,第二个是邮箱,第三和第四是密码(输入时没有任何显示,实际上已经输入了)。最后显示创建成功!

Username (leave blank to use 'root'): admin
Email address: admin@markhoo.com
Password:
Password (again):
Superuser created successfully.

启动 Django

(myblog_env) [root@instance-pkb22o0h myblog]# python manage.py runserver 0.0.0.0:8000

如果没有报错,就说明 Django 已经安装成功了,并且跟 Mysql 的连接正常。打开浏览器输入你的服务器IP:8000,就可以看到你的网站了

退出 Django

按 ctrl+c 退出 Django 服务

使用 Gunicorn

Gunicorn 一般用来管理多个进程,有进程挂了Gunicorn 可以把它拉起来,防止服务器长时间停止服务,还可以动态调整 worker 的数量,请求多的时候增加 worker 的数量,请求少的时候减少。

在虚拟环境下,安装 Gunicorn:

(myblog_env) [root@instance-pkb22o0h myblog]# pip3 install gunicorn

测试Gunicorn是否能启动你的项目服务

(myblog_env) [root@instance-pkb22o0h myblog]# gunicorn --bind 0.0.0.0:8000 myblog.wsgi:application

注意:myblog.wsgi:application这里的myblog对应的是 /home/markhoo/sites/markhoo.com/myblog/myblog, 根据自己的修改。浏览器输入域名,可以看到访问成功了!

访问ip地址看浏览器是否能正常查看内容(此时没有退出虚拟环境)
完成测试后,按CTRL-C 停止 Gunicorn 运行

退出虚拟环境

deactivate

创建一个 Gunicorn Systemd Service 文件,直接运行以下命令

vim /etc/systemd/system/gunicorn.service

修改内容如下:

[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=root
Group=nginx
WorkingDirectory=/home/markhoo/sites/markhoo.com/myblog
ExecStart=/home/markhoo/sites/markhoo.com/myblog_env/bin/gunicorn --workers 3 --bind unix:/home/markhoo/sites/markhoo.com/myblog/myblog.sock myblog.wsgi:application

[Install]
WantedBy=multi-user.target

一定要注意自己的项目路径和虚拟环境路径
WorkingDirectory与ExecStart 修改为自己的路径

开启Gunicorn服务并开机自启,运行以下命令

systemctl start gunicorn
systemctl enable gunicorn

配置 Nginx

接下是配置 Nginx 来处理用户请求

配置nginx代理通过Gunicorn
依次运行以下命令

(myblog_env) [root@instance-pkb22o0h myblog]# cd /etc/nginx/
(myblog_env) [root@instance-pkb22o0h nginx]# ls
conf.d fastcgi.conf fastcgi_params koi-utf mime.types nginx.conf scgi_params uwsgi
default.d fastcgi.conf.default fastcgi_params.default koi-win mime.types.default nginx.conf.default scgi_params.default uwsgi
(myblog_env) [root@instance-pkb22o0h nginx]# vi nginx.conf

或者直接运行

vim /etc/nginx/nginx.conf

打开后默认的内容如下:

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

找到server内容进行修改,具体内容如下:

server {
    listen 80;
    server_name  markhoo.com 0.0.0.0;

        location = /favicon.ico { access_log off; log_not_found off; }
        location /static/ {
        root /home/markhoo/sites/markhoo.com/myblog;
        }

        location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://unix:/home/markhoo/sites/markhoo.com/myblog/myblog.sock;
        }

    }

注意修改自己的IP地址或域名,还有文件路径
server_domain_or_IP 代表你的IP地址或域名

  • 这里的域名为 markhoo.com,其次一定要添加 0.0.0.0 否则无法访问,两个之间用空格隔开。
  • 所有URL 带有 /static 的请求均由 Nginx 处理,alias 指明了静态文件的存放目录。
  • 其它请求转发给 Django 处理。proxy_pass 后面使用了 unix 套接字,其作用是防止端口冲突,这里就不再详述。

修改所涉及到的vim命令使用方法:使用 'vi (文件名称)' 是打开文件,然后点击键盘 'i' 键切换到输入模式。修改完需要退出vim编辑器,先按键盘 'Esc' 键,然后输入 ':wq' 保存退出。如果不想修改又无法退出可以用 ':q!' 强制退出,这样不会保存你修改的内容。
更多的vi、vim命令的具体使用方法可以自行到搜索引擎搜索学习,这里也不一一讲解了。

修改nginx的权限

usermod -a -G root nginx
chmod 710 /home/
nginx -t

如果没有报错,就行下一步操作

开启nginx服务并开机自启

systemctl start nginx
systemctl enable nginx

现在,一切配置完成!你可以访问你的域名了!

官网太慢甚至打不开,所以干脆转载转来,方便学习。

|xadmin| 插件制作

插件原理

|xadmin| 的插件系统架构设计一定程度上借鉴了 wordpress 的设计。 想要了解 |xadmin| 的插件系统架构首先需要了解 |xadmin| AdminView 的概念。
简单来说,就是 |xadmin| 系统中每一个页面都是一个 AdminView 对象返回的 HttpResponse 结果。|xadmin| 的插件系统做的事情其实就是在 AdminView
运行过程中改变其执行的逻辑, 或是改变其返回的结果,起到修改或增强原有功能的效果。下面让我们看看整个插件从制作完成到实际运行的整个过程。

首先需要创建自己的插件类, 插件类继承 :class:~xadmin.views.BaseAdminPlugin ::


class HelloWorldPlugin(BaseAdminPlugin):
    ...

开发好的插件首先要注册到 |xadmin| 中, 示例代码如下::


# ListAdminView 是 Model 列表页面
xadmin.site.register_plugin(HelloWorldPlugin, ListAdminView)

其中插件的注册和使用可以参看 :meth:xadmin.sites.AdminSite.register_plugin

当将插件注册到 |xadmin| 后, |xadmin| 在创建 AdminView 实例的时候会将该插件放入实例的 :attr:plugins 属性。当 AdminView 在处理请求
时,会首先逐个调用 :attr:plugins 中插件的 :meth:~xadmin.views.BaseAdminPlugin.init_request 方法,插件在该方法中一般进行初始化的操作并且返回一个 Boolean 值告诉 AdminView
是否需要加载该插件。当 :meth:~xadmin.views.BaseAdminPlugin.init_request 方法返回值为 False 时, AdminView 不会加载该插件。实例如下::


class HelloWorldPlugin(BaseAdminPlugin):
    say_hello = False
    # 初始化方法根据 ``say_hello`` 属性值返回
    def init_request(self, *args, **kwargs):
        return bool(self.say_hello)

在以上实例中,插件根据自身的 say_hello 属性来决定是否让自己被加载。您可能会迷惑, say_hello 属性看起来一直会是 False 呀,那样这个插件不是永远不会被加载?
其实 |xadmin| 在创建插件实例的时候会将 OptionClass 的同名属性替换插件的属性。这样,在不同的 OptionClass 下会有不同的插件结果,实例如下::


class SomeModelAdmin(object):
    say_hello = True
    ...
site.register(SomeModel, SomeModelAdmin)

理解以上内容后,让我们再看看插件具体是如何起作用的。在 AdminView 的执行过程中,可以被插件截获或修改的方法使用 :func:~xadmin.views.base.filter_hook 装饰,实例如下::

class ListAdminView(ModelAdminView):
    # 可以被插件截获或修改的方法使用该装饰器装饰
    @filter_hook
    def get_context(self):
        ...

使用 :func:~xadmin.views.base.filter_hook 装饰的方法执行过程中会根据一定原则执行插件中的同名方法,具体信息查考该装饰器的文档内容。

.. autofunction:: xadmin.views.base.filter_hook

根据该装饰器的执行原则,如果我们想修改上面示例中 ListAdminViewget_context 的返回值,可以在插件中实现如下代码::


class HelloWorldPlugin(BaseAdminPlugin):
    # 在插件中加入同名方法,修改 ``ListAdminView`` 的 ``get_context`` 返回的值
    def get_context(self, context):
        context.update({'hello_target': 'World!!'})
        return context

如果我们希望插件在 AdminView 的方法前执行,或是完全使用自己的方法替代 AdminView 的方法可以这样::


class HelloWorldPlugin(BaseAdminPlugin):
    # 第一个参数为 ``__`` 。这样 ``__`` 即为 ``ListAdminView`` 的 ``get_context`` 方法本身,注意,这时还没有执行这个方法。
    def get_context(self, __):
        context = {'hello_target': 'World!!'}
        #我们可以在任何时候执行 ``AdminView`` 的方法,或是根本不执行
        context.update(__())
        return context

至此,加入的插件就实现了对 AdminView 方法的完全控制。

模板插件

我们知道,Django 中一个完整的 View 是包含模板的,模板用来生成 View 最终返回的 HTML 内容。当然,插件也可以在模板中插入自己的内容。我们来看看具体如何实现。

首先让我们来看看 |xadmin| 中的模板代码示例片段 (change_list.html):

.. code-block:: html


{% load xadmin %}
...
<form id="changelist-form" action="" method="post"{% view_block 'result_list_form' %}>{% csrf_token %}
  {% view_block 'results_top' %}
  <div class="results">
    {% if results %}
    ...

其中的 view_block Tag 即为插件的 插入点 。插件可以在自己的插件类中使用 block_ + 插入点名称 方法将 HTML 片段插入到页面的这个位置,示例如下:

.. code-block:: python


class HelloWorldPlugin(BaseAdminPlugin):
    
    # context 即为 TemplateContext, nodes 参数包含了其他插件的返回内容。
    # 您可以直接返回 HTML 片段,或是将内容加入到 nodes 参数中
    def block_results_top(self, context, nodes):
        return s"<div class='info'>Hello %s</div>" % context['hello_target']

插件实例

下面让我们来看一个 |xadmin| 中完整的插件实例::


from django.template import loader

from xadmin.sites import site
from xadmin.views import BaseAdminPlugin, ListAdminView

REFRESH_VAR = '_refresh'

# 该插件实现了一个列表页面刷新器的效果
class RefreshPlugin(BaseAdminPlugin):

    # 用户可以定制刷新的频率,可以传入多个值。该属性会被 ``OptionClass`` 的同名属性替换
    refresh_times = []
    
    def init_request(self, *args, **kwargs):
        # 根据用户是否制定了刷新器来决定是否启动该插件
        return bool(self.refresh_times)

    # 插件拦截了返回 Media 的方法,加入自己需要的 js 文件。
    def get_media(self, media):
        if self.request.GET.get(REFRESH_VAR):
            # 放页面处于自动刷新状态时,加入自己的 js 制定刷新逻辑
            media.add_js([self.static('xadmin/js/refresh.js')])
        return media

    # Block Views
    # 在页面中插入 HTML 片段,显示刷新选项。
    def block_top_toolbar(self, context, nodes):
        current_refresh = self.request.GET.get(REFRESH_VAR)
        context.update({
            'has_refresh': bool(current_refresh),
            'clean_refresh_url': self.admin_view.get_query_string(remove=(REFRESH_VAR,)),
            'current_refresh': current_refresh,
            'refresh_times': [{
                'time': r,
                'url': self.admin_view.get_query_string({REFRESH_VAR: r}),
                'selected': str(r) == current_refresh,
            } for r in self.refresh_times],
        })
        # 可以将 HTML 片段加入 nodes 参数中
        nodes.append(loader.render_to_string('xadmin/blocks/refresh.html', context_instance=context))
# 注册插件
site.register_plugin(RefreshPlugin, ListAdminView)

最后不要忘记在适当的地方加载该代码, 让其执行。一般情况下,你可以将其写到 adminx.py 文件中,这样,只要您的 APP 加入到 Django Settings 的 INSTALL_APPS 中,
|xadmin| 就会自动执行 app 下的 adminx.py 文件。

插件开发

了解了插件的运行原理后我们就可以开发自己的插件了。首先我们需要了解插件类中的实用方法。因为插件是继承 :class:~xadmin.views.BaseAdminPlugin 类,而该类继承自
:class:~xadmin.views.BaseAdminObject,所以这两个类的方法都可以在插件中使用。

|xadmin| 在创建插件时会自动注入以下属性到插件实例中:


* request : Http Request

* user : 当前 User 对象

* args : View 方法的 args 参数

* kwargs : View 方法的 kwargs 参数

* admin_view : ``AdminView`` 实例

* admin_site : |xadmin| 的 ``admin_site`` 对象实例

如果 AdminView 是 :class:~xadmin.views.ModelAdminView 的子类,还会自动注入以下属性:


* model : Model 对象

* opts : Model 的 _meta 属性

接下来您应该考虑打算制作什么功能的插件了。不同功能的插件额能需要注册到不同的 AdminView 上,|xadmin| 系统中
主要的 AdminView 有:


* :class:`~xadmin.views.BaseAdminView` : 所有 ``AdminView`` 的基础类,注册在该 View 上的插件可以影响所有的 ``AdminView``

* :class:`~xadmin.views.CommAdminView` : 用户已经登陆后显示的 View,也是所有登陆后 View 的基础类。该 View主要作用是创建了 |xadmin| 的通用元素,例如:系统菜单,用户信息等。插件可以通过注册该 View 来修改这些信息。

* :class:`~xadmin.views.ModelAdminView` : 基于 Model 的 ``AdminView`` 的基础类,注册的插件可以影响所有基于 Model 的 View。

* :class:`~xadmin.views.ListAdminView` : Model 列表页面 View。

* :class:`~xadmin.views.ModelFormAdminView` : Model 编辑页面 View。

* :class:`~xadmin.views.CreateAdminView` : Model 创建页面 View。

* :class:`~xadmin.views.UpdateAdminView` : Model 修改页面 View。

* :class:`~xadmin.views.DeleteAdminView` : Model 删除页面 View。

* :class:`~xadmin.views.DetailAdminView` : Model 详情页面 View。

选择好目标 AdminView 后就要在自己的插件中编写方法来修改或增强这些 AdminView 。其中每个 AdminView 可以
拦截的方法及其介绍请参看各 AdminView 的文档。

插件规范

文档模板::

"""
Name
======

作者
----

该插件的作者信息

功能
----

描述插件的主要功能

截图
----

.. image:: /images/plugins/action.png

使用
----

描述插件的使用方法,  以及使用示例.

版本
----

描述插件的版本信息

API
---
.. autoclass:: XXX

"""

昨晚安装Django的时候总是安装到一半就报错,而且速度很慢,问题主要是国外的源连接不稳定会中断所致。后来就想到了切换到国内源。想到可能还会有同学碰到类似的问题,特此记录分享一下

pip国内的一些镜像

公网:http://e.pypi.python.org/simple
教育网:http://pypi.tuna.tsinghua.edu.cn/simple
清华大学 https://pypi.tuna.tsinghua.edu.cn/simple/
豆瓣(douban) http://pypi.douban.com/simple/
阿里云 http://mirrors.aliyun.com/pypi/simple/
中国科技大学 https://pypi.mirrors.ustc.edu.cn/simple/
华中理工大学http://pypi.hustunique.com/simple
山东理工大学http://pypi.sdutlinux.org/simple

修改源方法:

临时使用:

可以在使用pip的时候在后面加上-i参数,指定pip源

pip install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple

指定需要安装的库requirements.txt

pip install -r "/home/admin/$APP_NAME/conf/requirements.txt" -i "http://mirrors.aliyun.com/pypi/simple" --trusted-host "mirrors.aliyun.com" && #通过阿里云镜像安装三方库

永久修改:

linux:

修改 ~/.pip/pip.conf (没有就创建一个), 内容如下:

[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

windows:

直接在user目录中创建一个pip目录,如:C:Usersxxpip,新建文件pip.ini,内容如下

[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

参考:https://pip.pypa.io/en/latest/configuration/

精选的Python框架,库,软件和资源的精选列表。


管理员面板

管理界面的库

  • Ajenti - 您的服务器应得的管理面板。
  • django-suit - 替代Django管理界面(仅限非商业用途)。
  • django-xadmin - 直接替换Django管理员有很多好东西。
  • flask-admin - Flask的简单且可扩展的管理界面框架。
  • flower - Celery的实时监控和Web管理员。
  • Grappelli - Django Admin-Interface的爵士皮肤。
  • Wooey - 一个为Python脚本创建自动Web UI的Django应用程序。

算法和设计模式

Python实现算法和设计模式

  • algorithms - Python中数据结构和算法的最小示例
  • PyPattyrn - 一个简单而有效的库,用于实现通用设计模式。
  • python-patterns - Python中的一组设计模式。
  • sortedcontainers - SortedList,SortedDict和SortedSet类型的快速,纯Python实现。
  • SCOOP - Python中的可扩展并发操作。

反垃圾邮件

打击垃圾邮件的图书馆

资产管理

用于管理,压缩和缩小网站资产的工具。

音频

用于操纵音频的库

  • audiolazy - 用于Python的表达式数字信号处理(DSP)包。
  • audioread - 跨库(GStreamer + Core Audio + MAD + FFmpeg)音频解码。
  • beets - 音乐库管理器和MusicBrainz标记器。
  • dejavu - 音频指纹识别和识别。
  • django-elastic-transcoder - Django + Amazon Elastic Transcoder.
  • eyeD3 - 用于处理音频文件的工具,特别是包含ID3元数据的MP3文件。
  • id3reader - 用于读取MP3元数据的Python模块。
  • m3u8 - 用于解析m3u8文件的模块。
  • mingus - 具有MIDI文件和播放支持的高级音乐理论和乐谱包。
  • mutagen - 用于处理音频元数据的Python模块。
  • pyAudioAnalysis - Python音频分析库:特征提取,分类,分段和应用程序
  • pydub - 使用简单易用的高级界面处理音频。
  • pyechonest - Python client for the Echo Nest API.
  • talkbox - 用于语音/信号处理的Python库。
  • TimeSide - 开放式网络音频处理框架。
  • tinytag - 用于读取MP3,OGG,FLAC和Wave文件的音乐元数据的库。

认证

用于实现认证方案的库

  • OAuth

    • Authomatic - 简单但功能强大的框架无关的身份验证/授权客户端。
    • django-allauth - Django的认证应用程序“正常工作”。
    • django-oauth-toolkit - 用于Django的OAuth 2好东西。
    • Flask-OAuthlib - OAuth 1.0 / a,2.0实现Flask的客户端和提供程序。
    • OAuthLib - OAuth请求签名逻辑的通用且全面的实现。
    • python-oauth2 - 一个经过全面测试的抽象界面,用于创建OAuth客户端和服务器。
    • python-social-auth - 一种易于设置的社交认证机制。
    • rauth - OAuth 1.0 / a,2.0和Ofly的Python库。
    • sanction - 一个简单的OAuth2客户端实现。
  • 其他

    • jose - JavaScript对象签名和加密草案实现。
    • PyJWT - JSON Web令牌草案01的实现。
    • python-jws - JSON Web签名草案02的实现。
    • python-jwt - 用于生成和验证JSON Web令牌的模块。

构建工具

从源代码编译软件

  • BitBake - 嵌入式Linux的类似make的构建工具。
  • buildout - 用于从多个部分创建,组装和部署应用程序的构建系统。
  • PlatformIO - 用于构建具有不同开发平台的代码的控制台工具。
  • PyBuilder - 用纯Python编写的连续构建工具。
  • SCons - 一种软件构建工具。

内置类增强功能

用于增强Python内置类的库

  • attrs - Replacement for __init__, __eq__, __repr__, etc. boilerplate in class definitions.
  • bidict - Efficient, Pythonic bidirectional map data structures and related functionality..
  • Box - Python dictionaries with advanced dot notation access.
  • dotted - A library that provides a method of accessing lists and dicts with a dotted path notation.

CMS

Content Management Systems.

  • django-cms - An Open source enterprise CMS based on the Django.
  • djedi-cms - A lightweight but yet powerful Django CMS with plugins, inline editing and performance in mind.
  • FeinCMS - One of the most advanced Content Management Systems built on Django.
  • Kotti - A high-level, Pythonic web application framework built on Pyramid.
  • Mezzanine - A powerful, consistent, and flexible content management platform.
  • Opps - A Django-based CMS for magazines, newspapers websites and portals with high-traffic.
  • Plone - A CMS built on top of the open source application server Zope.
  • Quokka - Flexible, extensible, small CMS powered by Flask and MongoDB.
  • Wagtail - A Django content management system.
  • Widgy - Last CMS framework, based on Django.

高速缓存

Libraries for caching data.

  • Beaker - A library for caching and sessions for use with web applications and stand-alone Python scripts and applications.
  • DiskCache - SQLite and file backed cache backend with faster lookups than memcached and redis.
  • django-cache-machine - Automatic caching and invalidation for Django models.
  • django-cacheops - A slick ORM cache with automatic granular event-driven invalidation.
  • django-viewlet - Render template parts with extended cache control.
  • dogpile.cache - dogpile.cache is next generation replacement for Beaker made by same authors.
  • HermesCache - Python caching library with tag-based invalidation and dogpile effect prevention.
  • johnny-cache - A caching framework for Django applications.
  • pylibmc - A Python wrapper around the libmemcached interface.

ChatOps工具

Libraries for chatbot development.

  • Errbot - The easiest and most popular chatbot to implement ChatOps.

代码分析

Tools of static analysis, linters and code quality checkers. See: awesome-static-analysis.

  • Code Analysis

    • flake8 - A wrapper around pycodestyle, pyflakes and McCabe.
    • coala - Language independent and easily extendable code analysis application.
    • code2flow - Turn your Python and JavaScript code into DOT flowcharts.
    • prospector - A tool to analyse Python code.
    • pycallgraph - A library that visualises the flow (call graph) of your Python application.
    • pylama - A code audit tool for Python and JavaScript.
    • pylint - A fully customizable source code analyzer.
  • Static Type Checkers

    • mypy - Check variable types during compile time.
    • Pyre - Performant type checking.
  • Static Type Annotations Generators

    • MonkeyType - A system for Python that generates static type annotations by collecting runtime types

命令行工具

Libraries for building command-line application.

  • Command-line Application Development

    • asciimatics - A package to create full-screen text UIs (from interactive forms to ASCII animations).
    • bashplotlib - Making basic plots in the terminal.
    • cement - CLI Application Framework for Python.
    • click - A package for creating beautiful command line interfaces in a composable way.
    • cliff - A framework for creating command-line programs with multi-level commands.
    • clint - Python Command-line Application Tools.
    • colorama - Cross-platform colored terminal text.
    • docopt - Pythonic command line arguments parser.
    • Gooey - Turn command line programs into a full GUI application with one line
    • Python-Fire - A library for creating command line interfaces from absolutely any Python object.
    • python-prompt-toolkit - A library for building powerful interactive command lines.
  • Productivity Tools

    • aws-cli - A universal command-line interface for Amazon Web Services.
    • cookiecutter - A command-line utility that creates projects from cookiecutters (project templates).
    • doitlive - A tool for live presentations in the terminal.
    • howdoi - Instant coding answers via the command line.
    • httpie - A command line HTTP client, a user-friendly cURL replacement.
    • kube-shell - An integrated shell for working with the Kubernetes CLI.
    • mycli - A Terminal Client for MySQL with AutoCompletion and Syntax Highlighting.
    • PathPicker - Select files out of bash output.
    • percol - Adds flavor of interactive selection to the traditional pipe concept on UNIX.
    • pgcli - Postgres CLI with autocompletion and syntax highlighting.
    • SAWS - A Supercharged AWS CLI.
    • thefuck - Correcting your previous console command.
    • tmuxp - A tmux session manager.
    • try - A dead simple CLI to try out python packages - it's never been easier.

兼容性

Libraries for migrating from Python 2 to 3.

  • Python-Future - The missing compatibility layer between Python 2 and Python 3.
  • Python-Modernize - Modernizes Python code for eventual Python 3 migration.
  • Six - Python 2 and 3 compatibility utilities.

计算机视觉

Libraries for computer vision.

并发和并行

用于并发和并行执行的库。

  • eventlet - Asynchronous framework with WSGI support.
  • gevent - A coroutine-based Python networking library that uses greenlet.
  • multiprocessing - (Python standard library) Process-based "threading" interface.
  • threading - (Python standard library) Higher-level threading interface.
  • Tomorrow - Magic decorator syntax for asynchronous code.
  • uvloop - Ultra fast implementation of asyncio event loop on top of libuv.

组态

用于存储和解析配置选项的库。

  • config - Hierarchical config from the author of logging.
  • ConfigObj - INI file parser with validation.
  • ConfigParser - (Python standard library) INI file parser.
  • profig - Config from multiple formats with value conversion.
  • python-decouple - Strict separation of settings from code.

加密

  • cryptography - A package designed to expose cryptographic primitives and recipes to Python developers.
  • hashids - Implementation of hashids in Python.
  • Paramiko - A Python (2.6+, 3.3+) implementation of the SSHv2 protocol, providing both client and server functionality.
  • Passlib - Secure password storage/hashing library, very high level.
  • PyNacl - Python binding to the Networking and Cryptography (NaCl) library.

数据分析

用于数据分析的库。

  • Blaze - NumPy和Pandas与大数据的接口。
  • Open Mining - Pandas界面中的商业智能(BI)。
  • Orange - 通过可视化编程或脚本进行数据挖掘,数据可视化,分析和机器学习。
  • Pandas - 提供高性能,易于使用的数据结构和数据分析工具的库。
  • Optimus - 使用PySpark后端进行清理,预处理,功能工程,探索性数据分析和简单的机器学习。

数据验证

用于验证数据的库。在许多情况下用于表单。

  • Cerberus - 一个轻量级和可扩展的数据验证库。
  • colander - Validating and deserializing data obtained via XML, JSON, an HTML form post.
  • jsonschema - An implementation of JSON Schema for Python.
  • schema - 用于验证Python数据结构的库。
  • Schematics - Data Structure Validation.
  • valideer - Lightweight extensible data validation and adaptation library.
  • voluptuous - A Python data validation library.

数据可视化

用于可视化数据的库。请参阅: awesome-javascript.

  • Altair - Declarative statistical visualization library for Python.
  • Bokeh - Python的交互式Web绘图。
  • bqplot - Interactive Plotting Library for the Jupyter Notebook
  • ggplot - Same API as ggplot2 for R.
  • Matplotlib - Python 2D绘图库。
  • Pygal - A Python SVG Charts Creator.
  • PyGraphviz - Python interface to Graphviz.
  • PyQtGraph - 交互式和实时2D / 3D /图像绘图和科学/工程小部件。
  • Seaborn - 使用Matplotlib进行统计数据可视化。
  • VisPy - 基于OpenGL的高性能科学可视化。

数据库

Databases implemented in Python.

  • pickleDB - A simple and lightweight key-value store for Python.
  • TinyDB - A tiny, document-oriented database.
  • ZODB - A native object database for Python. A key-value and object graph database.

数据库驱动

用于连接和操作数据库的库

  • MySQL - awesome-mysql

    • mysqlclient - MySQL connector with Python 3 support (mysql-python fork).
    • oursql - A better MySQL connector with support for native prepared statements and BLOBs.
    • PyMySQL - A pure Python MySQL driver compatible to mysql-python.
  • PostgreSQL - awesome-postgres

    • psycopg2 - The most popular PostgreSQL adapter for Python.
    • queries - A wrapper of the psycopg2 library for interacting with PostgreSQL.
    • txpostgres - Twisted based asynchronous driver for PostgreSQL.
  • Other Relational Databases

    • apsw - Another Python SQLite wrapper.
    • dataset - Store Python dicts in a database - works with SQLite, MySQL, and PostgreSQL.
    • pymssql - A simple database interface to Microsoft SQL Server.
  • NoSQL Databases

    • cassandra-python-driver - Python driver for Cassandra.
    • HappyBase - A developer-friendly library for Apache HBase.
    • Plyvel - A fast and feature-rich Python interface to LevelDB.
    • py2neo - Python wrapper client for Neo4j's restful interface.
    • pycassa - Python Thrift driver for Cassandra.
    • PyMongo - The official Python client for MongoDB.
    • redis-py - The Redis Python Client.
    • telephus - Twisted based client for Cassandra.
    • txRedis - Twisted based client for Redis.

日期和时间

Libraries for working with dates and times.

  • Chronyk - 一个Python 3库,用于解析人类编写的时间和日期。
  • dateutil - Extensions to the standard Python datetime module.
  • delorean - A library for clearing up the inconvenient truths that arise dealing with datetimes.
  • moment - A Python library for dealing with dates/times. Inspired by Moment.js.
  • Pendulum - Python datetimes made easy.
  • PyTime - A easy-use Python module which aims to operate date/time/datetime by string.
  • pytz - World timezone definitions, modern and historical. Brings the tz database into Python.
  • when.py - Providing user-friendly functions to help perform common date and time actions.
  • maya - Datetimes for Humans, Maya is mostly built around the headaches and use-cases around parsing datetime data from websites.

调试工具

Libraries for debugging code.

  • pdb-like Debugger

    • ipdb - IPython-enabled pdb.
    • pdb++ - Another drop-in replacement for pdb.
    • pudb - A full-screen, console-based Python debugger.
    • remote-pdb - Remote vanilla PDB (over TCP sockets).
    • wdb - An improbable web debugger through WebSockets.
  • Profiler

    • line_profiler - Line-by-line profiling.
    • memory_profiler - Monitor Memory usage of Python code.
    • Pyflame - Attach this Ptracing Profiler to any processes running Python. Perfect for profiling production webservers.
    • profiling - An interactive Python profiler.
    • vprof - Visual Python profiler.
  • Others

    • IceCream - Inspect variables, expressions, and program execution with a single, simple function call.
    • django-debug-toolbar - Display various debug information for Django.
    • django-devserver - A drop-in replacement for Django's runserver.
    • flask-debugtoolbar - A port of the django-debug-toolbar to flask.
    • hunter - Hunter is a flexible code tracing toolkit.
    • lptrace - strace for Python programs.
    • manhole - Debug service that will accept unix domain socket connections and present the stacktraces for all threads and an interactive prompt.
    • pyelftools - Parsing and analyzing ELF files and DWARF debugging information.
    • pyringe - Debugger capable of attaching to and injecting code into Python processes.

深度学习

神经网络和深度学习的框架。请参阅: awesome-deep-learning.

  • Caffe - A fast open framework for deep learning..
  • Keras - 一个高级神经网络库,能够在TensorFlow或Theano之上运行。
  • MXNet - A deep learning framework designed for both efficiency and flexibility.
  • Neupy - Running and testing different Artificial Neural Networks algorithms.
  • Pytorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration.
  • Serpent.AI - Game agent framework. Use any video game as a deep learning sandbox.
  • TensorFlow - Google创建的最受欢迎的深度学习框架。
  • Theano - A library for fast numerical computation.

DevOps工具

DevOps的软件和库

  • Ansible - A radically simple IT automation platform.
  • Cloud-Init - A multi-distribution package that handles early initialization of a cloud instance.
  • cuisine - Chef-like functionality for Fabric.
  • Docker Compose - Fast, isolated development environments using Docker.
  • Fabric - A simple, Pythonic tool for remote execution and deployment.
  • Fabtools - Tools for writing awesome Fabric files.
  • honcho - A Python clone of Foreman, for managing Procfile-based applications.
  • OpenStack - Open source software for building private and public clouds.
  • pexpect - Controlling interactive programs in a pseudo-terminal like GNU expect.
  • psutil - A cross-platform process and system utilities module.
  • SaltStack - Infrastructure automation and management system.
  • supervisor - Supervisor process control system for UNIX.

Distribution

用于创建发布分发的打包可执行文件的库

  • dh-virtualenv - Build and distribute a virtualenv as a Debian package.
  • Nuitka - Compile scripts, modules, packages to an executable or extension module.
  • py2app - Freezes Python scripts (Mac OS X).
  • py2exe - Freezes Python scripts (Windows).
  • PyInstaller - Converts Python programs into stand-alone executables (cross-platform).
  • pynsist - A tool to build Windows installers, installers bundle Python itself.

文档

用于生成项目文档的库

  • Sphinx - Python文档生成器。

  • MkDocs - Markdown友好文档生成器。
  • pdoc - Epydoc replacement to auto generate API documentation for Python libraries.
  • Pycco - The literate-programming-style documentation generator.

下载

Libraries for downloading.

  • s3cmd - A command line tool for managing Amazon S3 and CloudFront.
  • s4cmd - Super S3 command line tool, good for higher performance.
  • you-get - A YouTube/Youku/Niconico video downloader written in Python 3.
  • youtube-dl - A small command-line program to download videos from YouTube.

电子商务

用于电子商务和支付的框架和库

  • alipay - 用于Python的非官方支付宝API。
  • Cartridge - A shopping cart app built using the Mezzanine.
  • django-oscar - An open-source e-commerce framework for Django.
  • django-shop - 基于Django的商店系统。
  • merchant - A Django app to accept payments from various payment processors.
  • money - Money class with optional CLDR-backed locale-aware formatting and an extensible currency exchange solution.
  • python-currencies - Display money format and its filthy currencies.
  • forex-python - 外汇汇率,比特币价格指数和货币转换。
  • saleor - An e-commerce storefront for Django.
  • shoop - 基于Django的开源电子商务平台。

编辑器插件和IDE

  • Emacs

    • Elpy - Emacs Python Development Environment.
  • Sublime Text

    • Anaconda - Anaconda turns your Sublime Text 3 in a full featured Python development IDE.
    • SublimeJEDI - A Sublime Text plugin to the awesome auto-complete library Jedi.
  • Vim

    • Jedi-vim - Vim bindings for the Jedi auto-completion library for Python.
    • Python-mode - 用于将Vim转换为Python IDE的一体化插件。
    • YouCompleteMe - Includes Jedi-based completion engine for Python.
  • Visual Studio

    • PTVS - Python Tools for Visual Studio.
  • Visual Studio Code

    • Python - An extension with rich support for the Python language, with features including linting, IntelliSense, formatting, refactoring, debugging, unit tests, and jupyter support.
    • Magic Python - Cutting edge Python syntax highlighter for Sublime Text, Atom, and Visual Studio Code. Used by GitHub to highlight your Python code!
  • IDE

    • LiClipse - Free polyglot IDE based on Eclipse. Uses PyDev for Python support.
    • PyCharm - Commercial Python IDE by JetBrains. Has free community edition available.
    • Spyder - Open Source Python IDE.

电子邮件

用于发送和解析电子邮件的库

  • envelopes - Mailing for human beings.
  • flanker - A email address and Mime parsing library.
  • imbox - Python IMAP for Humans.
  • inbox.py - Python SMTP Server for Humans.
  • lamson - Pythonic SMTP Application Server.
  • Marrow Mailer - High-performance extensible mail delivery framework.
  • modoboa - A mail hosting and management platform including a modern and simplified Web UI.
  • Nylas Sync Engine - Providing a RESTful API on top of a powerful email sync platform.
  • yagmail - Yet another Gmail/SMTP client.

环境管理

用于Python版本和环境管理的库

  • Pipenv - Sacred Marriage of Pipfile, Pip, & Virtualenv.
  • p - Dead simple interactive Python version management.
  • pyenv - Simple Python version management.
  • venv - (Python standard library in Python 3.3+) Creating lightweight virtual environments.
  • virtualenv - 创建独立Python环境的工具。
  • virtualenvwrapper - A set of extensions to virtualenv.

Files

用于文件操作和MIME类型检测的库

  • imghdr -(Python标准库)确定图像的类型。
  • mimetypes - (Python标准库)将文件名映射到MIME类型。
  • path.py - A module wrapper for os.path.
  • pathlib - (Python standard library in Python 3.4+) An cross-platform, object-oriented path library.
  • python-magic - A Python interface to the libmagic file type identification library.
  • Unipath - 面向对象的文件/目录操作方法。
  • watchdog - API and shell utilities to monitor file system events.

外部函数接口

用于提供外部函数接口的库

  • cffi - Python的外部函数接口,用于调用C代码。
  • ctypes - (Python standard library) Foreign Function Interface for Python calling C code.
  • PyCUDA - A Python wrapper for Nvidia's CUDA API.
  • SWIG - Simplified Wrapper and Interface Generator.

Forms

用于处理表单的库

  • Deform - Python HTML form generation library influenced by the formish form generation library.
  • django-bootstrap3 - Bootstrap 3 integration with Django.
  • django-crispy-forms - A Django app which lets you create beautiful forms in a very elegant and DRY way.
  • django-remote-forms - A platform independent Django form serializer.
  • WTForms - 灵活的表单验证和呈现库。

Functional Programming

Functional Programming with Python.

  • CyToolz - Cython implementation of Toolz: High performance functional utilities.
  • fn.py - Functional programming in Python: implementation of missing features to enjoy FP.
  • funcy - A fancy and practical functional tools.
  • Toolz - A collection of functional utilities for iterators, functions, and dictionaries.

GUI

用于处理图形用户界面应用程序的库

  • curses - Built-in wrapper for ncurses used to create terminal GUI applications.
  • Eel - Little library for making simple Electron-like offline HTML/JS GUI apps, with full access to Python capabilities and libraries.
  • enaml - Creating beautiful user-interfaces with Declaratic Syntax like QML.
  • Flexx - Flexx is a pure Python toolkit for creating GUI's, that uses web technology for its rendering.
  • kivy - A library for creating NUI applications, running on Windows, Linux, Mac OS X, Android and iOS.
  • pyglet - A cross-platform windowing and multimedia library for Python.
  • PyGObject - Python Bindings for GLib/GObject/GIO/GTK+ (GTK+3).
  • PyQt - Python bindings for the Qt cross-platform application and UI framework, with support for both Qt v4 and Qt v5 frameworks.
  • PySide - Python bindings for the Qt cross-platform application and UI framework, supporting the Qt v4 framework.
  • pywebview - A lightweight cross-platform native wrapper around a webview component that allows to display HTML content in its own native dedicated window.
  • Tkinter - Tkinter is Python's de-facto standard GUI package.
  • Toga - A Python native, OS native GUI toolkit.
  • urwid - A library for creating terminal GUI applications with strong support for widgets, events, rich colors, etc.
  • wxPython - A blending of the wxWidgets C++ class library with the Python.

游戏开发

令人敬畏的游戏开发库

  • Cocos2d - cocos2d is a framework for building 2D games, demos, and other graphical/interactive applications. It is based on pyglet.
  • Panda3D - 3D game engine developed by Disney and maintained by Carnegie Mellon's Entertainment Technology Center. Written in C++, completely wrapped in Python.
  • Pygame - Pygame is a set of Python modules designed for writing games.
  • PyOgre - Python bindings for the Ogre 3D render engine, can be used for games, simulations, anything 3D.
  • PyOpenGL - Python ctypes bindings for OpenGL and it's related APIs.
  • PySDL2 - A ctypes based wrapper for the SDL2 library.
  • RenPy - A Visual Novel engine.

地理位置

Libraries for geocoding addresses and working with latitudes and longitudes.

  • django-countries - A Django app that provides country choices for use with forms, flag icons static files, and a country field for models.
  • GeoDjango - 世界级的地理网络框架。
  • GeoIP - Python API for MaxMind GeoIP Legacy Database.
  • geojson - Python bindings and utilities for GeoJSON.
  • geopy - Python Geocoding Toolbox.
  • pygeoip - Pure Python GeoIP API.

HTML Manipulation

用于处理HTML和XML的库

  • BeautifulSoup - Providing Pythonic idioms for iterating, searching, and modifying HTML or XML.
  • bleach - A whitelist-based HTML sanitization and text linkification library.
  • cssutils - A CSS library for Python.
  • html5lib - A standards-compliant library for parsing and serializing HTML documents and fragments.
  • lxml - A very fast, easy-to-use and versatile library for handling HTML and XML.
  • MarkupSafe - Implements a XML/HTML/XHTML Markup safe string for Python.
  • pyquery - A jQuery-like library for parsing HTML.
  • untangle - Converts XML documents to Python objects for easy access.
  • WeasyPrint - A visual rendering engine for HTML and CSS that can export to PDF.
  • xmldataset - Simple XML Parsing.
  • xmltodict - Working with XML feel like you are working with JSON.

HTTP

用于处理HTTP的库

  • grequests - requests + gevent for asynchronous HTTP requests.
  • httplib2 - Comprehensive HTTP client library.
  • requests - HTTP Requests for Humans™.
  • treq - Python requests like API built on top of Twisted's HTTP client.
  • urllib3 - A HTTP library with thread-safe connection pooling, file post support, sanity friendly.

Hardware

Libraries for programming with hardware.

  • ino - Command line toolkit for working with Arduino.
  • keyboard - Hook and simulate global keyboard events on Windows and Linux.
  • mouse - Hook and simulate global mouse events on Windows and Linux.
  • Pingo - Pingo provides a uniform API to program devices like the Raspberry Pi, pcDuino, Intel Galileo, etc.
  • PyUserInput - A module for cross-platform control of the mouse and keyboard.
  • scapy - A brilliant packet manipulation library.
  • wifi - 用于在Linux上使用WiFi的Python库和命令行工具。

图像处理

用于处理图像的库

  • hmap - Image histogram remapping.
  • imgSeek - A project for searching a collection of images using visual similarity.
  • nude.py - Nudity detection.
  • pagan - Retro identicon (Avatar) generation based on input string and hash.
  • pillow - Pillow is the friendly PIL fork.
  • pyBarcode - Create barcodes in Python without needing PIL.
  • pygram - Instagram-like image filters.
  • python-qrcode - A pure Python QR Code generator.
  • Quads - Computer art based on quadtrees.
  • scikit-image - A Python library for (scientific) image processing.
  • thumbor - A smart imaging service. It enables on-demand crop, re-sizing and flipping of images.
  • wand - Python bindings for MagickWand, C API for ImageMagick.

Implementations

Implementations of Python.

  • CLPython - Implementation of the Python programming language written in Common Lisp.
  • CPython - Default, most widely used implementation of the Python programming language written in C.
  • Cython - Optimizing Static Compiler for Python. Uses type mixins to compile Python into C or C++ modules resulting in large performance gains
  • Grumpy - More compiler than interpreter as more powerful CPython2.7 replacement (alpha).
  • IronPython - Implementation of the Python programming language written in C# targeting the .NET Framework and Mono.
  • Jython - Implementation of Python programming language written in Java for the Java virtual machine (JVM).
  • MicroPython - MicroPython - a lean and efficient Python programming language implementation for microcontrollers and constrained systems
  • Numba - Python JIT compiler to LLVM aimed at scientific Python.
  • PeachPy - x86-64 assembler embedded in Python. Can be used as inline assembler for Python or as a stand-alone assembler for Windows, Linux, OS X, Native Client and Go.
  • Pyjion - A JIT for Python based upon CoreCLR.
  • PyPy - Implementation of the Python programming language written in RPython and translated into C. PyPy focuses on speed, efficiency and compatibility with the original CPython interpreter. The interpreter uses black magic to make Python very fast without having to add in additional type information.
  • PySec - Hardened version of python that makes it easier for security professionals and developers to write applications more resilient to attacks and manipulations.
  • Pyston - A Python implementation built using LLVM and modern JIT techniques with the goal of achieving good performance.
  • Stackless Python - An enhanced version of the Python programming language which allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads.

Interactive Interpreter

Interactive Python interpreters (REPL).

国际化

Libraries for working with i18n.

  • Babel - Python的国际化库。
  • PyICU - A wrapper of International Components for Unicode C++ library (ICU).

Job Scheduler

Libraries for scheduling jobs.

  • APScheduler - A light but powerful in-process task scheduler that lets you schedule functions.
  • django-schedule - A calendaring app for Django.
  • doit - A task runner and build tool.
  • gunnery - Multipurpose task execution tool for distributed systems with web-based interface.
  • Joblib - A set of tools to provide lightweight pipelining in Python.
  • Plan - Writing crontab file in Python like a charm.
  • schedule - Python job scheduling for humans.
  • Spiff - A powerful workflow engine implemented in pure Python.
  • TaskFlow - A Python library that helps to make task execution easy, consistent and reliable.

Logging

Libraries for generating and working with logs.

  • Eliot - Logging for complex & distributed systems.
  • logbook - Logging replacement for Python.
  • logging - (Python standard library) Logging facility for Python.
  • raven - Python client for Sentry, a log/error tracking, crash reporting and aggregation platform for web applications.

Machine Learning

Libraries for Machine Learning. See: awesome-machine-learning.

  • H2O - Open Source Fast Scalable Machine Learning Platform.
  • Metrics - Machine learning evaluation metrics.
  • NuPIC - Numenta Platform for Intelligent Computing.
  • scikit-learn - The most popular Python library for Machine Learning.
  • Spark ML - Apache Spark's scalable Machine Learning library.
  • vowpal_porpoise - A lightweight Python wrapper for Vowpal Wabbit.
  • xgboost - A scalable, portable, and distributed gradient boosting library.

MapReduce

Frameworks and libraries for MapReduce.

  • PySpark - Apache Spark Python API.
  • luigi - A module that helps you build complex pipelines of batch jobs.
  • mrjob - Run MapReduce jobs on Hadoop or Amazon Web Services.
  • streamparse - Run Python code against real-time streams of data. Integrates with Apache Storm.
  • dask - A flexible parallel computing library for analytic computing.

Microsoft Windows

Python programming on Microsoft Windows.

  • Python(x,y) - Scientific-applications-oriented Python Distribution based on Qt and Spyder.
  • pythonlibs - Unofficial Windows binaries for Python extension packages.
  • PythonNet - Python Integration with the .NET Common Language Runtime (CLR).
  • PyWin32 - Python Extensions for Windows.
  • WinPython - Portable development environment for Windows 7/8.

Miscellaneous

Useful libraries or tools that don't fit in the categories above.

  • blinker - A fast Python in-process signal/event dispatching system.
  • itsdangerous - Various helpers to pass trusted data to untrusted environments.
  • pluginbase - A simple but flexible plugin system for Python.
  • Pychievements - A framework for creating and tracking achievements.
  • Tryton - A general purpose business framework.

Natural Language Processing

Libraries for working with human languages.

  • gensim - Topic Modelling for Humans.
  • Jieba - Chinese text segmentation.
  • langid.py - Stand-alone language identification system.
  • NLTK - A leading platform for building Python programs to work with human language data.
  • Pattern - A web mining module for the Python.
  • polyglot - Natural language pipeline supporting hundreds of languages.
  • SnowNLP - A library for processing Chinese text.
  • spaCy - A library for industrial-strength natural language processing in Python and Cython.
  • TextBlob - Providing a consistent API for diving into common NLP tasks.
  • PyTorch-NLP - A toolkit enabling rapid deep learning NLP prototyping for research.

Network Virtualization

Tools and libraries for Virtual Networking and SDN (Software Defined Networking).

  • Mininet - A popular network emulator and API written in Python.
  • POX - An open source development platform for Python-based Software Defined Networking (SDN) control applications, such as OpenFlow SDN controllers.
  • Pyretic - A member of the Frenetic family of SDN programming languages that provides powerful abstractions over network switches or emulators.
  • SDX Platform - SDN based IXP implementation that leverages Mininet, POX and Pyretic.

Networking

用于网络编程的库

  • asyncio - (Python standard library) Asynchronous I/O, event loop, coroutines and tasks.
  • diesel - Greenlet-based event I/O Framework for Python.
  • pulsar - Event-driven concurrent framework for Python.
  • pyzmq - A Python wrapper for the ZeroMQ message library.
  • Twisted - An event-driven networking engine.
  • txZMQ - Twisted based wrapper for the ZeroMQ message library.
  • NAPALM - Cross-vendor API to manipulate network devices.

News Feed

用于构建用户活动的库

ORM

Libraries that implement Object-Relational Mapping or data mapping techniques.

  • Relational Databases

    • Django Models - A part of Django.
    • SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper.

    • Orator - The Orator ORM provides a simple yet beautiful ActiveRecord implementation.
    • Peewee - A small, expressive ORM.
    • PonyORM - ORM that provides a generator-oriented interface to SQL.
    • pyDAL - A pure Python Database Abstraction Layer.
    • python-sql - Write SQL queries pythonically.
  • NoSQL Databases

  • Others

    • butterdb - A Python ORM for Google Drive Spreadsheets.
    • dataset - A JSON-based database.

Package Management

Libraries for package and dependency management.

  • pip - The Python package and dependency manager.

  • conda - Cross-platform, Python-agnostic binary package manager.
  • Curdling - Curdling is a command line tool for managing Python packages.
  • pip-tools - A set of tools to keep your pinned Python dependencies fresh.
  • wheel - The new standard of Python distribution and are intended to replace eggs.

Package Repositories

Local PyPI repository server and proxies.

  • warehouse - Next generation Python Package Repository (PyPI).

  • bandersnatch - PyPI mirroring tool provided by Python Packaging Authority (PyPA).
  • devpi - PyPI server and packaging/testing/release tool.
  • localshop - Local PyPI server (custom packages and auto-mirroring of pypi).

Permissions

Libraries that allow or deny users access to data or functionality.

  • Carteblanche - Module to align code with thoughts of users and designers. Also magically handles navigation and permissions.
  • django-guardian - Implementation of per object permissions for Django 1.2+
  • django-rules - A tiny but powerful app providing object-level permissions to Django, without requiring a database.

Processes

Libraries for starting and communicating with OS processes.

Queue

Libraries for working with event and task queues.

  • celery - An asynchronous task queue/job queue based on distributed message passing.
  • huey - Little multi-threaded task queue.
  • mrq - Mr. Queue - A distributed worker task queue in Python using Redis & gevent.
  • rq - Simple job queues for Python.
  • simpleq - A simple, infinitely scalable, Amazon SQS based queue.

Recommender Systems

Libraries for building recommender systems.

  • annoy - Approximate Nearest Neighbors in C++/Python optimized for memory usage.
  • fastFM - A library for Factorization Machines.
  • implicit - A fast Python implementation of collaborative filtering for implicit datasets.
  • libffm - A library for Field-aware Factorization Machine (FFM).
  • LightFM - A Python implementation of a number of popular recommendation algorithms.
  • Spotlight - Deep recommender models using PyTorch.
  • surprise - A scikit for building and analyzing recommender systems.
  • TensorRec - A Recommendation Engine Framework in TensorFlow.

RESTful API

Libraries for developing RESTful APIs.

  • Django

  • Flask

    • eve - REST API framework powered by Flask, MongoDB and good intentions.
    • flask-api-utils - Taking care of API representation and authentication for Flask.
    • flask-api - Browsable Web APIs for Flask.
    • flask-restful - Quickly building REST APIs for Flask.
    • flask-restless - Generating RESTful APIs for database models defined with SQLAlchemy.
  • Pyramid

    • cornice - A RESTful framework for Pyramid.
  • Framework agnostic

    • falcon - A high-performance framework for building cloud APIs and web app backends.
    • hug - A Python3 framework for cleanly exposing APIs over HTTP and the Command Line with automatic documentation and validation.
    • restless - Framework agnostic REST framework based on lessons learned from Tastypie.
    • ripozo - Quickly creating REST/HATEOAS/Hypermedia APIs.
    • sandman - Automated REST APIs for existing database-driven systems.
    • apistar - A smart Web API framework, designed for Python 3.

RPC Servers

RPC-compatible servers.

Science

Libraries for scientific computing.

  • astropy - A community Python library for Astronomy.
  • bcbio-nextgen - Providing best-practice pipelines for fully automated high throughput sequencing analysis.
  • bccb - Collection of useful code related to biological analysis.
  • Biopython - Biopython is a set of freely available tools for biological computation.
  • cclib - A library for parsing and interpreting the results of computational chemistry packages.
  • Colour - A colour science package implementing a comprehensive number of colour theory transformations and algorithms.
  • NetworkX - A high-productivity software for complex networks.
  • NIPY - A collection of neuroimaging toolkits.
  • NumPy - A fundamental package for scientific computing with Python.
  • Open Babel - A chemical toolbox designed to speak the many languages of chemical data.
  • ObsPy - A Python toolbox for seismology.
  • PyDy - Short for Python Dynamics, used to assist with workflow in the modeling of dynamic motion.
  • PyMC - Markov Chain Monte Carlo sampling toolkit.
  • QuTiP - Quantum Toolbox in Python.
  • RDKit - Cheminformatics and Machine Learning Software.
  • SciPy - A Python-based ecosystem of open-source software for mathematics, science, and engineering.
  • statsmodels - Statistical modeling and econometrics in Python.
  • SymPy - A Python library for symbolic mathematics.
  • Zipline - A Pythonic algorithmic trading library.
  • SimPy - A process-based discrete-event simulation framework.

Search

Libraries and software for indexing and performing search queries on data.

Serialization

Libraries for serializing complex data types

  • marshmallow - marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, to and from native Python datatypes.

Serverless Frameworks

Frameworks for developing serverless Python code.

  • apex - Build, deploy, and manage AWS Lambda functions with ease.
  • python-lambda - A toolkit for developing and deploying Python code in AWS Lambda.
  • Zappa - A tool for deploying WSGI applications on AWS Lambda and API Gateway.

Specific Formats Processing

Libraries for parsing and manipulating specific text formats.

  • General

    • tablib - A module for Tabular Datasets in XLS, CSV, JSON, YAML.
  • Office

    • Marmir - Takes Python data structures and turns them into spreadsheets.
    • openpyxl - A library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files.
    • pyexcel - Providing one API for reading, manipulating and writing csv, ods, xls, xlsx and xlsm files.
    • python-docx - Reads, queries and modifies Microsoft Word 2007/2008 docx files.
    • python-pptx - Python library for creating and updating PowerPoint (.pptx) files.
    • relatorio - Templating OpenDocument files.
    • unoconv - Convert between any document format supported by LibreOffice/OpenOffice.
    • XlsxWriter - A Python module for creating Excel .xlsx files.
    • xlwings - A BSD-licensed library that makes it easy to call Python from Excel and vice versa.
    • xlwt / xlrd - Writing and reading data and formatting information from Excel files.
  • PDF

    • PDFMiner - A tool for extracting information from PDF documents.
    • PyPDF2 - A library capable of splitting, merging and transforming PDF pages.
    • ReportLab - Allowing Rapid creation of rich PDF documents.
  • Markdown

    • Mistune - Fastest and full featured pure Python parsers of Markdown.
    • Python-Markdown - A Python implementation of John Gruber’s Markdown.
  • YAML

    • PyYAML - YAML implementations for Python.
  • CSV

    • csvkit - Utilities for converting to and working with CSV.
  • Archive

    • unp - A command line tool that can unpack archives easily.

Static Site Generator

Static site generator is a software that takes some text + templates as input and produces HTML files on the output.

  • Cactus - Static site generator for designers.
  • Hyde - Jinja2-based static web site generator.
  • Lektor - An easy to use static CMS and blog engine.
  • Nikola - A static website and blog generator.
  • Pelican - Uses Markdown or ReST for content and Jinja 2 for themes. Supports DVCS, Disqus. AGPL.
  • Tinkerer - Tinkerer is a blogging engine/.static website generator powered by Sphinx.

Tagging

Libraries for tagging items.

Template Engine

Libraries and tools for templating and lexing.

  • Genshi - Python templating toolkit for generation of web-aware output.
  • Jinja2 - A modern and designer friendly templating language.
  • Mako - Hyperfast and lightweight templating for the Python platform.

Testing

Libraries for testing codebases and generating test data.

  • Testing Frameworks

    • hypothesis - Hypothesis is an advanced Quickcheck style property based testing library.
    • mamba - The definitive testing tool for Python. Born under the banner of BDD.
    • nose - A nicer unittest for Python.
    • nose2 - The successor to nose, based on unittest2.
    • pytest - A mature full-featured Python testing tool.
    • Robot Framework - A generic test automation framework.
    • unittest - (Python standard library) Unit testing framework.
  • Test Runners

    • green - A clean, colorful test runner.
    • tox - Auto builds and tests distributions in multiple Python versions
  • GUI / Web Testing

    • locust - Scalable user load testing tool written in Python.
    • PyAutoGUI - PyAutoGUI is a cross-platform GUI automation Python module for human beings.
    • Selenium - Python bindings for Selenium WebDriver.
    • sixpack - A language-agnostic A/B Testing framework.
    • splinter - Open source tool for testing web applications.
  • Mock

    • doublex - Powerful test doubles framework for Python.
    • freezegun - Travel through time by mocking the datetime module.
    • httmock - A mocking library for requests for Python 2.6+ and 3.2+.
    • httpretty - HTTP request mock tool for Python.
    • mock - (Python standard library) A mocking and patching library.
    • Mocket - Socket Mock Framework plus HTTP[S]/asyncio/gevent mocking library with recording/replaying capability.
    • responses - A utility library for mocking out the requests Python library.
    • VCR.py - Record and replay HTTP interactions on your tests.
  • Object Factories

    • factory_boy - A test fixtures replacement for Python.
    • mixer - Another fixtures replacement. Supported Django, Flask, SQLAlchemy, Peewee and etc.
    • model_mommy - Creating random fixtures for testing in Django.
  • Code Coverage

  • Fake Data

    • mimesis - is a Python library that help you generate fake data.
    • fake2db - Fake database generator.
    • faker - A Python package that generates fake data.
    • radar - Generate random datetime / time.
  • Error Handler

    • FuckIt.py - FuckIt.py uses state-of-the-art technology to make sure your Python code runs whether it has any right to or not.

Text Processing

Libraries for parsing and manipulating plain texts.

  • General

    • chardet - Python 2/3 compatible character encoding detector.
    • difflib - (Python standard library) Helpers for computing deltas.
    • ftfy - Makes Unicode text less broken and more consistent automagically.
    • fuzzywuzzy - Fuzzy String Matching.
    • Levenshtein - Fast computation of Levenshtein distance and string similarity.
    • pangu.py - Spacing texts for CJK and alphanumerics.
    • pyfiglet - An implementation of figlet written in Python.
    • pypinyin - Convert Chinese hanzi to pinyin.
    • shortuuid - A generator library for concise, unambiguous and URL-safe UUIDs.
    • textdistance - Compute distance between sequences. 30+ algorithms, pure python implementation, common interface, optional external libs usage.
    • unidecode - ASCII transliterations of Unicode text.
    • uniout - Print readable chars instead of the escaped string.
    • xpinyin - A library to translate Chinese hanzi (漢字) to pinyin (拼音).
  • Slugify

    • awesome-slugify - A Python slugify library that can preserve unicode.
    • python-slugify - A Python slugify library that translates unicode to ASCII.
    • unicode-slugify - A slugifier that generates unicode slugs with Django as a dependency.
  • Parser

    • phonenumbers - Parsing, formatting, storing and validating international phone numbers.
    • PLY - Implementation of lex and yacc parsing tools for Python.
    • Pygments - A generic syntax highlighter.
    • pyparsing - A general purpose framework for generating parsers.
    • python-nameparser - Parsing human names into their individual components.
    • python-user-agents - Browser user agent parser.
    • sqlparse - A non-validating SQL parser.

Third-party APIs

Libraries for accessing third party services APIs. See: List of Python API Wrappers and Libraries.

URL Manipulation

Libraries for parsing URLs.

  • furl - A small Python library that makes parsing and manipulating URLs easy.
  • purl - A simple, immutable URL class with a clean API for interrogation and manipulation.
  • pyshorteners - A pure Python URL shortening lib.
  • short_url - Python implementation for generating Tiny URL and bit.ly-like URLs.
  • webargs - A friendly library for parsing HTTP request arguments, with built-in support for popular web frameworks, including Flask, Django, Bottle, Tornado, and Pyramid.

Video

Libraries for manipulating video and GIFs.

  • moviepy - A module for script-based movie editing with many formats, including animated GIFs.
  • scikit-video - Video processing routines for SciPy.

WSGI Servers

WSGI-compatible web servers.

  • bjoern - Asynchronous, very fast and written in C.
  • fapws3 - Asynchronous (network side only), written in C.
  • gunicorn - Pre-forked, partly written in C.
  • meinheld - Asynchronous, partly written in C.
  • netius - Asynchronous, very fast.
  • rocket - Multi-threaded.
  • uWSGI - A project aims at developing a full stack for building hosting services, written in C.
  • waitress - Multi-threaded, powers Pyramid.
  • Werkzeug - A WSGI utility library for Python that powers Flask and can easily be embedded into your own projects.

Web Content Extracting

Libraries for extracting web contents.

  • Haul - An Extensible Image Crawler.
  • html2text - Convert HTML to Markdown-formatted text.
  • lassie - Web Content Retrieval for Humans.
  • micawber - A small library for extracting rich content from URLs.
  • newspaper - News extraction, article extraction and content curation in Python.
  • python-goose - HTML Content/Article Extractor.
  • python-readability - Fast Python port of arc90's readability tool.
  • requests-html - Pythonic HTML Parsing for Humans.
  • sanitize - Bringing sanity to world of messed-up data.
  • sumy - A module for automatic summarization of text documents and HTML pages.
  • textract - Extract text from any document, Word, PowerPoint, PDFs, etc.
  • toapi - Every web site provides APIs.

Web Crawling & Web Scraping

Libraries to automate data extraction from websites.

  • cola - A distributed crawling framework.
  • Demiurge - PyQuery-based scraping micro-framework.
  • feedparser - Universal feed parser.
  • Grab - Site scraping framework.
  • MechanicalSoup - A Python library for automating interaction with websites.
  • portia - Visual scraping for Scrapy.
  • pyspider - A powerful spider system.
  • RoboBrowser - A simple, Pythonic library for browsing the web without a standalone web browser.
  • Scrapy - A fast high-level screen scraping and web crawling framework.

Web Frameworks

Full stack web frameworks.

WebSocket

Libraries for working with WebSocket.

Services

Online tools and APIs to simplify development.

Continuous Integration

See: awesome-CIandCD.

  • CircleCI - A CI service that can run very fast parallel testing. (GitHub only)
  • Travis CI - A popular CI service for your open source and private projects. (GitHub only)
  • Vexor CI - A continuous integration tool for private apps with pay-per-minute billing model.
  • Wercker - A Docker-based platform for building and deploying applications and microservices.

Code Quality

  • Codacy - Automated Code Review to ship better code, faster.
  • Codecov - Code coverage dashboard.
  • CodeFactor - Automated Code Review for Git.
  • Landscape - Hosted continuous Python code metrics.

Resources

Where to discover new Python libraries.

Podcasts

Twitter

Websites

Weekly

Other Awesome Lists

List of lists.

Contributing

Your contributions are always welcome! Please take a look at the contribution guidelines first.

I will keep some pull requests open if I'm not sure whether those libraries are awesome, you could vote for them by adding :+1: to them. Pull requests will be merged when their votes reach 20.


If you have any question about this opinionated list, do not hesitate to contact me @vinta on Twitter or open an issue on GitHub.