学习分享|初入Anaconda3以及实践

2023-11-27 19:38:26 浏览数 (1)

简介: Anaconda,中文大蟒蛇,是一个开源的Python发行版本,其包含了conda、Python等180多个科学包及其依赖项。

Anaconda3

Anaconda,中文大蟒蛇,是一个开源的Python发行版本,其包含了conda、Python等180多个科学包及其依赖项。我当初安装这个是为了学习人工智能和算法。

1、安装

1、命令操作

安装好之后,可以使用命令conda,可以使用一系列命令参数,conda --help 或 conda -h 、 conda --version 或conda -V

例如,使用conda info命令如下所示信息:

代码语言:javascript复制
$xxx:~ xxx$ conda info
     active environment : None
            shell level : 0
       user config file : /Users/xxx/.condarc
 populated config files : /Users/xxx/.condarc
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __osx=10.15.7
       base environment : /Users/xxx/opt/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                         https://repo.anaconda.com/pkgs/main/noarch
                         https://repo.anaconda.com/pkgs/r/osx-64
                         https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/xxx/opt/anaconda3/pkgs
                          /Users/xxx/.conda/pkgs
       envs directories : /Users/xxx/opt/anaconda3/envs
                          /Users/xxx/.conda/envs
               platform : osx-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Darwin/19.6.0 OSX/10.15.7
                UID:GID : 501:20
             netrc file : None
           offline mode : False

2、导出所有存在依赖

1)导出所有依赖包到记事本

conda list -e > [fileName(.txt)]

例如,conda list -e > /usr/local/data/conda-lib-98.txt

1、Kafka 集群防火墙

2、运行情况,或者执行命令,查看status

3、手动执行消费者或生产者,订阅消息,查看消费情况

问题记录FQA

Q1、ModuleNotFoundError: No module named 'pymysql'

Q1:执行python程序,出现 ``ModuleNotFoundError: No module named 'pymysql'

A1:

1)查找确认依赖库

初步判定是安装阿naconda3缺少pymysql依赖包,执行命令`` conda list `` 分别查看85服务器和96服务器anaconda3的已经安装插件库,依赖包如下所示,85服务器已安装插件库295个插件,96服务器已安装插件库288个插件

2)查找、下载依赖库

85插件所在路径, /data/_software/_install_pack/_python/,anaconda3的home路径,/data/_software/python/anaconda3/

86插件所在路径,, /data/_software/_pack/_python/,anaconda3的home路径,/data/_software/python/anaconda3/

96插件所在路径,/data/_software/_package/,anaconda3的home路径,/data/_software/python/anaconda3/

经过插件列表比对并排查,发现96服务器缺失插件列表如下所示

85服务器有whl插件源件:

  • PyNaCl-1.3.0-cp34-abi3-manylinux1_x86_64.whl
  • bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl
  • confluent_kafka-1.1.0-cp37-cp37m-manylinux1_x86_64.whl
  • paramiko-2.6.0-py2.py3-none-any.whl
  • PyMySQL-0.9.3-py2.py3-none-any.whl
  • cx_Oracle-8.0.0-cp37-cp37m-manylinux1_x86_64.whl

85服务器未找到whl插件源件(其中avro-python3使用python安装,cx-oracle使用pip安装):

  • avro-python3=1.8.2=pypi_0
  • cx-oracle=8.0.0=pypi_0

先下载所需要的插件包,whl格式本质上是一个压缩包,里面包含了py文件,以及经过编译的pyd文件,使得可以在不具备编译环境的情况下,选择合适自己的python环境进行安装,简单来说,适合内网环境。

其次有些时候在内网环境中并不可以使用pip命令来下载依赖包,这就出现了需要手动安装依赖包的情况,由于网络不通,采取手动安装插件库,查找85服务器是否存在插件库或网络下载插件库再传给96服务器。经过检查,发现原来85服务器已经下载插件。在85服务器找到所依赖的插件,打包上传到96服务器。

3)安装依赖库

安装插件的命令使用pip,此命令在已经内置在anaconda3的依赖插件中,具体位置为home/lib/python3.7/site-packages。

  • 安装已有whl文件

安装bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl

代码语言:javascript复制
$ pip install bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl
Processing ./bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl
Requirement already satisfied: cffi>=1.1 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from bcrypt==3.1.7) (1.12.3)
Requirement already satisfied: six>=1.4.1 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from bcrypt==3.1.7) (1.12.0)
Requirement already satisfied: pycparser in /data/_software/python/anaconda3/lib/python3.7/site-packages (from cffi>=1.1->bcrypt==3.1.7) (2.19)
Installing collected packages: bcrypt
Successfully installed bcrypt-3.1.7
$ pip install confluent_kafka-1.1.0-cp37-cp37m-manylinux1_x86_64.whl
Processing ./confluent_kafka-1.1.0-cp37-cp37m-manylinux1_x86_64.whl
Installing collected packages: confluent-kafka
Successfully installed confluent-kafka-1.1.0

安装PyNaCl-1.3.0-cp34-abi3-manylinux1_x86_64.whl

先安装PyNaCl,再安装paramiko

代码语言:javascript复制
$ pip install PyNaCl-1.3.0-cp34-abi3-manylinux1_x86_64.whl
Processing ./PyNaCl-1.3.0-cp34-abi3-manylinux1_x86_64.whl
Requirement already satisfied: cffi>=1.4.1 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from PyNaCl==1.3.0) (1.12.3)
Requirement already satisfied: six in /data/_software/python/anaconda3/lib/python3.7/site-packages (from PyNaCl==1.3.0) (1.12.0)
Requirement already satisfied: pycparser in /data/_software/python/anaconda3/lib/python3.7/site-packages (from cffi>=1.4.1->PyNaCl==1.3.0) (2.19)
Installing collected packages: PyNaCl
Successfully installed PyNaCl-1.3.0

安装paramiko-2.6.0-py2.py3-none-any.whl

代码语言:javascript复制
$ pip install paramiko-2.6.0-py2.py3-none-any.whl
Processing ./paramiko-2.6.0-py2.py3-none-any.whl
Requirement already satisfied: bcrypt>=3.1.3 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from paramiko==2.6.0) (3.1.7)
Requirement already satisfied: pynacl>=1.0.1 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from paramiko==2.6.0) (1.3.0)
Requirement already satisfied: cryptography>=2.5 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from paramiko==2.6.0) (2.7)
Requirement already satisfied: six>=1.4.1 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from bcrypt>=3.1.3->paramiko==2.6.0) (1.12.0)
Requirement already satisfied: cffi>=1.1 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from bcrypt>=3.1.3->paramiko==2.6.0) (1.12.3)
Requirement already satisfied: asn1crypto>=0.21.0 in /data/_software/python/anaconda3/lib/python3.7/site-packages (from cryptography>=2.5->paramiko==2.6.0) (0.24.0)
Requirement already satisfied: pycparser in /data/_software/python/anaconda3/lib/python3.7/site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko==2.6.0) (2.19)
Installing collected packages: paramiko
Successfully installed paramiko-2.6.0

安装PyMySQL-0.9.3-py2.py3-none-any.whl

代码语言:javascript复制
$ pip install PyMySQL-0.9.3-py2.py3-none-any.whl
Processing ./PyMySQL-0.9.3-py2.py3-none-any.whl
Installing collected packages: PyMySQL
Successfully installed PyMySQL-0.9.3

安装cx_Oracle-8.0.0-cp37-cp37m-manylinux1_x86_64.whl

代码语言:javascript复制
$ pip install cx_Oracle-8.0.0-cp37-cp37m-manylinux1_x86_64.whl
Processing ./cx_Oracle-8.0.0-cp37-cp37m-manylinux1_x86_64.whl
Installing collected packages: cx-Oracle
Successfully installed cx-Oracle-8.0.0

安装未有whl文件的tar.gz文件,进入到arvo_python3所在目录

代码语言:javascript复制
cd /data/_software/_package

解压arvo_python3-1.8.2.tar.gz,会自动创建并解压到arvo_python3-1.8.2

代码语言:javascript复制
tar -zxvf arvo_python3-1.8.2.tar.gz

进入arvo_python3-1.8.2包

代码语言:javascript复制
cd arvo_python3-1.8.2

执行命令python setup.py install

代码语言:javascript复制
$ python setup.py install
running install
running bdist_egg
running egg_info
writing avro_python3.egg-info/PKG-INFO
writing dependency_links to avro_python3.egg-info/dependency_links.txt
writing top-level names to avro_python3.egg-info/top_level.txt
reading manifest file 'avro_python3.egg-info/SOURCES.txt'
writing manifest file 'avro_python3.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/avro
copying avro/__init__.py -> build/lib/avro
copying avro/datafile.py -> build/lib/avro
copying avro/io.py -> build/lib/avro
copying avro/ipc.py -> build/lib/avro
copying avro/protocol.py -> build/lib/avro
copying avro/schema.py -> build/lib/avro
copying avro/tool.py -> build/lib/avro
copying avro/txipc.py -> build/lib/avro
copying avro/HandshakeRequest.avsc -> build/lib/avro
copying avro/HandshakeResponse.avsc -> build/lib/avro
copying avro/VERSION.txt -> build/lib/avro
copying avro/LICENSE -> build/lib/avro
copying avro/NOTICE -> build/lib/avro
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/__init__.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/datafile.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/io.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/ipc.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/protocol.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/schema.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/tool.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/txipc.py -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/HandshakeRequest.avsc -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/HandshakeResponse.avsc -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/VERSION.txt -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/LICENSE -> build/bdist.linux-x86_64/egg/avro
copying build/lib/avro/NOTICE -> build/bdist.linux-x86_64/egg/avro
byte-compiling build/bdist.linux-x86_64/egg/avro/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/datafile.py to datafile.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/io.py to io.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/ipc.py to ipc.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/protocol.py to protocol.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/schema.py to schema.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/tool.py to tool.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/avro/txipc.py to txipc.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
installing scripts to build/bdist.linux-x86_64/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
creating build/scripts-3.7
copying and adjusting scripts/avro -> build/scripts-3.7
changing mode of build/scripts-3.7/avro from 600 to 755
creating build/bdist.linux-x86_64/egg/EGG-INFO/scripts
copying build/scripts-3.7/avro -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts
changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/avro to 755
copying avro_python3.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying avro_python3.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying avro_python3.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying avro_python3.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
avro.__pycache__.__init__.cpython-37: module references __file__
avro.__pycache__.ipc.cpython-37: module references __file__
creating dist
creating 'dist/avro_python3-1.8.2-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing avro_python3-1.8.2-py3.7.egg
creating /data/_software/python/anaconda3/lib/python3.7/site-packages/avro_python3-1.8.2-py3.7.egg
Extracting avro_python3-1.8.2-py3.7.egg to /data/_software/python/anaconda3/lib/python3.7/site-packages
Adding avro-python3 1.8.2 to easy-install.pth file
Installing avro script to /data/_software/python/anaconda3/bin
Installed /data/_software/python/anaconda3/lib/python3.7/site-packages/avro_python3-1.8.2-py3.7.egg
Processing dependencies for avro-python3==1.8.2
Finished processing dependencies for avro-python3==1.8.2

4)检查确认依赖库

执行安装新插件后,96服务器插件列表为295个插件

执行命令 conda list -e > [输出插件列表到文件.txt]

Q2、sshpass命令没有找到

Q2:执行python程序,提示sshpass命令没有找到

A2:

1)下载 sshpass-1.06.tar.gz

有网络情况下,执行命令

代码语言:javascript复制
wget http://sourceforge.net/projects/sshpass/files/latest/download -O sshpass.tar.gz

2)解压

解压sshpass-1.06.tar.gz到当前目录(自动创建目录并写入解压文件)

代码语言:javascript复制
tar -zxvf sshpass-1.06.tar.gz

3)编译

进入sshpass-1.06目录,进行编译

代码语言:javascript复制
./configure

4)安装

执行安装命令

代码语言:javascript复制
sudo make install

5)其他参考

https://linux.cn/article-8086-1.html

Q3、No such file or directory

Q3、执行python程序,控制台打印出来以下错误信息

代码语言:javascript复制
Traceback (most recent call last):
  File "test_oracle_lyc.py", line 4, in <module>
    db = cx_Oracle.connect("xxx.xxx.xxx.xxx", "username", "password", "orclName", charset='utf8' )
cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "libnsl.so.1: cannot open shared object file: No such file or directory". See https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html for help

A3:

1)python、oracle依赖包

经过排查,发现缺少oracle_client,安装oracle-instantclient19.3-basic-19.3.0.0.0-1.x86_64.rpm

2)安装

使用root用户执行安装命令,安装到默认路径

代码语言:javascript复制
rpm -Uvh oracle-instantclient19.3-basic-19.3.0.0.0-1.x86_64.rpm

3)环境变量

使用root用户登录进去配置相关环境变量(这个好像不用配置也可以,我在官方网站看到18 以下数据库客户端才需要配置环境变量,以及手动配置lib库地址和bin目录,现在我们使用的版本为19,已经自动创建libaio以及其他依赖包)。vim /etc/profile,输入环境变量信息

代码语言:javascript复制
export ORACLE_HOME=/usr/lib/oracle/19.3/client64
export ORABIN=/usr/lib/oracle/19.3/client64/bin
export LD_LIBRARY_PATH=/usr/lib/oracle/19.3/client64/lib
export PATH=$PATH:$ORACLE_HOME/bin

让环境变量立即生效

代码语言:javascript复制
source /etc/profile

4)安装系统依赖包

在root用户下,安装libnsl.so.1,进入到目录查看系统依赖包,由于我们都是使用的64bit操作系统,所有的软件工具也都是64bit

代码语言:javascript复制
cd /usr/lib64

查看(ls)是否有libnsl依赖包,在85服务器上,我查询到libnsl* 的依赖包为

代码语言:javascript复制
libnsl.so.2
libnsl.so.2.0.0

查看(ll)是否有libnsl依赖包,在85服务器上,我查询到libnsl* 的依赖包为

代码语言:javascript复制
lrwxrwxrwx     1 root root       15 Mar 15  2020  libnsl.so.2 -> libnsl.so.2.0.0
-rwxr-xr-x        1 root root 96480 Mar 15  2020  libnsl.so.2.0.0

然后进行拷贝,软连接设置

代码语言:javascript复制
cp -r libnsl.so.2 libnsl.so.1

再次执行查看操作

代码语言:javascript复制
lrwxrwxrwx     1 root root        15 Jan 23 16:40   libnsl.so.1 -> libnsl.so.2.0.0
lrwxrwxrwx     1 root root       15 Mar 15  2020  libnsl.so.2 -> libnsl.so.2.0.0
-rwxr-xr-x     1 root root 96480 Mar 15  2020  libnsl.so.2.0.0

5)切换测试

新打开一个窗口,使用appuser登录,或者重新登录到服务器,然后编写一个连接oracle的测试脚本,脚本如下

代码语言:javascript复制
import cx_Oracle
db  = connect("userName", "password", "IP:Port/serviceName", "UTF-8")
print(db.version)

保存代码为 test_oracle.py

执行python test_oracle.py,如果打印出来数据库服务器的版本号,则说明已经连接并成功读取数据。

总结

笔记属于个人的喜好,但是形成了自我的风格就是博文了。希望在记录自己博文道路越走越远。


我正在参与2023腾讯技术创作特训营第三期有奖征文,组队打卡瓜分大奖!

0 人点赞