搭建本地的PyPI仓库-devpi

2018-08-27 0 条评论 725 次阅读 0 人点赞

快乐的秘密不在于成就卓越,而在于不断成长。

0. 简介与背景

为什么要部署私有的PyPI?
1)官方的PyPI 没有收入来源,纯粹靠捐款,由Python软件基金会运行;
2)简化了共享代码的维护;
3)使用setuptools 将整个项目打包是非常好的做法;

devpi 主要有三部分组件:
1)devpi-server 提供总的仓库功能;
2)devpi-web 提供web界面以及search的功能;
3)devpi-client 命令行工具,提供私有包上传等功能;

部署环境 - Centos 6.8
因本人在做实验,所以使用pyenv+virtualenv组合方式,隔离出python 2.7.14的环境,部署devpi。读者大可直接在操作系统上部署devpi。

1. 部署devpi-server

1.1 简单部署

#1. 创建应用程序的隔离环境(直接安装在操作系统上的,可以跳过这一步)
pyenv virtualenv 2.7.14 devpi_2.7
pyenv activate devpi_2.7
#2. 安装 devpi-server
pip install -q -U devpi-server
(devpi_2.7) [root@devPython-182 ~]# pip list|grep devpi
devpi-common 3.2.2
devpi-server 4.4.0
(devpi_2.7) [root@devPython-182 ~]# devpi-server --version
4.4.0
#3. 初始化并启动 服务
(devpi_2.7) [root@devPython-182 ~]# devpi-server --init
(devpi_2.7) [root@devPython-182 ~]# devpi-server --start
2018-04-18 18:47:03,054 INFO NOCTX Loading node info from /root/.devpi/server/.nodeinfo
2018-04-18 18:47:03,056 INFO NOCTX wrote nodeinfo to: /root/.devpi/server/.nodeinfo
2018-04-18 18:47:03,059 WARNI NOCTX Can't open sqlite3 db with uri keyword. Python 3.4 is the first version to support it.
2018-04-18 18:47:03,060 WARNI NOCTX unable to open database file
2018-04-18 18:47:03,060 WARNI NOCTX Can't open sqlite3 db with options in URI. There is a higher possibility of read/write conflicts between threads, causing slowdowns due to retries.
starting background devpi-server at http://localhost:3141
/root/.devpi/server/.xproc/devpi-server$ /application/pyenv/versions/devpi_2.7/bin/devpi-server
process u'devpi-server' started pid=4685
devpi-server process startup detected
logfile is at /root/.devpi/server/.xproc/devpi-server/xprocess.log
#4. 客户机测试 /root/pypi/ 是固定路径
# ---> pip 测试
(first_project) [root@devPython-182 ~]# pip install -i http://localhost:3141/root/pypi/ simplejson
(first_project) [root@devPython-182 ~]# pip uninstall -y simplejson
# ---> easy_install 测试
(first_project) [root@devPython-182 ~]# easy_install -i http://localhost:3141/root/pypi/+simple/ simplejson

【总结】
使用以下任何一个地址,都是可以的:
1)http://localhost:3141/root/pypi/
2)http://localhost:3141/root/pypi/+simple/

1.2 部署search功能

在使用search前,请先安装 devpi-web(这个最好跟devpi-server一起安装),如果pip安装时出现错误,则可以先 devpi-server --stop,再安装。

pip install -q -U devpi-web
devpi-server --recreate-search-index  # 要等很久
devpi-server --stop
devpi-server --start
pip search --index http://localhost:3141/root/pypi/ devpi-client

1.3 指定发布的地址

# devpi-server 端启动时指定对外的访问IP和端口
devpi-server --start --host 10.0.0.182 --port 9001
# 客户端使用pip时如下,如不增加 trusted-host(可信赖站点),则会报错,要求部署要用HTTPS
pip install -i http://10.0.0.182:9001/root/pypi simplejson --trusted-host 10.0.0.182

1.4 客户端优化

修改~/.pip/pip.conf的值:

(first_project) [root@devPython-182 ~]# cat .pip/pip.conf
[global]
trusted-host=10.0.0.182
index-url=http://10.0.0.182:9001/root/pypi/

[search]
index = http://10.0.0.182:9001/root/pypi/

2. Package的操作

2.1 上传包

(secord_project) [root@devPython-182 hello]# tree
.
├── build
│ └── bdist.linux-x86_64
├── dist
│ └── hello_proj-1.0.0-py2.py3-none-any.whl
├── hello_proj.egg-info
│ ├── dependency_links.txt
│ ├── entry_points.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── hello.py
├── LICENSE.txt
├── README.md
├── setup.cfg
└── setup.py
# 安装devpi-client
(secord_project) [root@devPython-182 pylab]# pip install devpi-client
# 创建用户和新的Index
(secord_project) [root@devPython-182 pylab]# devpi use http://10.0.0.182:9001/root/pypi
(secord_project) [root@devPython-182 pylab]# devpi user -c dev01 password=123456
user created: dev01
(secord_project) [root@devPython-182 pylab]# devpi login dev01 --password=123456
logged in 'dev01', credentials valid for 10.00 hours
(secord_project) [root@devPython-182 pylab]# devpi index -c dev bases=root/pypi
http://10.0.0.182:9001/dev01/dev:
  type=stage
  bases=root/pypi
  volatile=True
  acl_upload=dev01
  acl_toxresult_upload=:ANONYMOUS:
  mirror_whitelist=
  pypi_whitelist=
(secord_project) [root@devPython-182 pylab]# devpi use dev01/dev
Warning: insecure http host, trusted-host will be set for pip
current devpi index: http://10.0.0.182:9001/dev01/dev (logged in as dev01)
venv for install/set commands: /application/pyenv/versions/3.5.5rc1/envs/secord_project
only setting venv pip cfg, no global configuration changed
/application/pyenv/versions/3.5.5rc1/envs/secord_project/pip.conf: no config file exists
always-set-cfg: no
# 上传
(secord_project) [root@devPython-182 pylab]# cd 3.5/hello/
(secord_project) [root@devPython-182 hello]# devpi upload
detected devpi:upload section in /pylab/3.5/hello/setup.cfg
using workdir /tmp/devpi0
pre-build: cleaning /pylab/3.5/hello/dist
--> /pylab/3.5/hello$ /application/pyenv/versions/secord_project/bin/python setup.py sdist --formats gztar 
built: /pylab/3.5/hello/dist/hello-proj-1.0.0.tar.gz [SDIST.TGZ] 2.504kb
register hello-proj-1.0.0 to http://10.0.0.182:9001/dev01/dev/
file_upload of hello-proj-1.0.0.tar.gz to http://10.0.0.182:9001/dev01/dev/

2.2 推送到仓库

当我们检查包没有问题后,就可以使用devpi push命令,把它推送到私有仓库或者PyPI上。

2.3 删除包

(secord_project) [root@devPython-182 hello]# devpi list hello-proj
http://10.0.0.182:9001/dev01/dev/+f/94d/2ecca817d776f/hello-proj-1.0.0.tar.gz
(secord_project) [root@devPython-182 hello]# devpi remove -y hello-proj
About to remove the following releases and distributions
version: 1.0.0
  - http://10.0.0.182:9001/dev01/dev/+f/94d/2ecca817d776f/hello-proj-1.0.0.tar.gz
Are you sure (yes/no)? yes (autoset from -y option)
deleting release 1.0.0 of hello-proj

3. devpi-server的操作

devpi-server --start  # 启动服务
devpi-server --stop # 停止服务
devpi-server --status # 检查服务状态
devpi-server --log #查看服务的运行日志

4. 永久运行devpi-server

上面的操作,我们是手工启动devpi-server,如果希望服务能随开机启动,我们有以下方案:
1)把启动的命令,加入系统启动文件中,例如: /etc/profile 文件、/etc/rc.local;
2)结合 supervisor 来创建devpi-server的后台守护进程;
3)定时任务、把devpi-server加入开机启动服务中;
以上的实现方式,不需要用户手工去创建了,devpi-server能生成对应的配置文件:

(devpi_2.7) [root@devPython-182 ~]# devpi-server --start --host 10.0.0.182 --port 3141 --serverdir /root/mydevpiserver
 --gen-config
wrote gen-config/supervisor-devpi.conf
wrote gen-config/nginx-devpi.conf
wrote gen-config/crontab
wrote gen-config/net.devpi.plist
wrote gen-config/devpi.service
查看一下devpi生成的配置文件
(devpi_2.7) [root@devPython-182 ~]# cd gen-config/
(devpi_2.7) [root@devPython-182 gen-config]# cat supervisor-devpi.conf 

[program:devpi-server]
command=/application/pyenv/versions/devpi_2.7/bin/devpi-server --port 3141 --host 10.0.0.182 --serverdir /root/mydevpiserver
user = root
priority = 999
startsecs = 5
redirect_stderr = True
autostart = True

4.1 supervisor

supervisor 是一个开源的进程管理工具。如想对它进行深入了解,可以查看我的另外一篇博文:supervisor进程管理工具

# 1. 安装
pip install supervisor
# 2. 生成配置文件
echo_supervisord_conf > /etc/supervisord.conf
# 3. 添加配置项
echo "[include]" >>/etc/supervisord.conf
echo "files = /etc/supervisor/conf.d/*.conf" >>/etc/supervisord.conf
# 4. 创建目录
mkdir -p /etc/supervisor/conf.d/
# 复制devpi-server对应的supervisor配置文件到该目录下,重启服务
(devpi_2.7) [root@devPython-182 gen-config]# cp supervisor-devpi.conf /etc/supervisor/conf.d/
(devpi_2.7) [root@devPython-182 gen-config]# ls /etc/supervisor/conf.d/
supervisor-devpi.conf
[root@devPython-182 tmp]# which supervisord
/application/pyenv/shims/supervisord
[root@devPython-182 tmp]# /application/pyenv/shims/supervisord -c /etc/supervisord.conf
[root@devPython-182 tmp]# /application/pyenv/shims/supervisorctl status all
devpi-server RUNNING pid 3318, uptime 0:00:48
[root@devPython-182 tmp]# netstat -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name   
tcp 0 0 10.0.0.182:3141 0.0.0.0:* LISTEN 3318/python2.7      
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1184/sshd           
tcp 0 0 :::22 :::* LISTEN 1184/sshd       

5. 安全

我们可用在启动服务时,使用--restrict-modify的选项来规定谁可用创建、修改和删除用户或者索引;

6. 高可用

https://devpi.net/docs/devpi/devpi/stable/+doc/adminman/server.html
https://devpi.net/docs/devpi/devpi/stable/+doc/adminman/replica.html

掌柜

让未来超越过去!

文章评论(0)