Keras.application 에는 ResNeXt101 등이 없음. Keras_application을 import 하면 존재하는데,
문제는 이걸 수행 시 아래와 같은 에러가 발생한다.
AttributeError: 'NoneType' object has no attribute 'image_data_format'
이의 해결책으로
import keras
keras_application.resnext.ResNeXt101(input_shape, input_tensor = None, include_top = False, weights = 'imagenet',
backend = keras.backend, layers = keras.layers, models = keras.models, utils = keras.utils)
를 사용하면 된다.
1. 맥에선 유독 pip install 또는 build에서 문제가 발생하는 현상
: colorcorrect 라이브러리 설치시 우분투, windows에서는 문제가 없는데 mac 에서만 pip 설치가 안됨.
[ Error message]
(tensorflow) iMac-Pro-mlsee21:colorcorrect-0.8 mslee21$ python setup.py build
/Users/mslee21/anaconda3/envs/tensorflow/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_requires'
warnings.warn(msg)
running build
running build_py
running build_ext
building 'colorcorrect._cutil' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mslee21/anaconda3/envs/tensorflow/include -arch x86_64 -I/Users/mslee21/anaconda3/envs/tensorflow/include -arch x86_64 -Icutil -I/Users/mslee21/anaconda3/envs/tensorflow/include/python3.6m -c cutil/cutil.cpp -o build/temp.macosx-10.7-x86_64-3.6/cutil/cutil.o
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on
the command line to use the libc++ standard library instead
[-Wstdlibcxx-not-found]
cutil/cutil.cpp:2:10: fatal error: 'vector' file not found
#include <vector>
^~~~~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
[ 해결과정 ] 솔루션을 찾아보니 xcode가 10으로 업그레드 하면서 libstdc++.tbd, libstdc++.6.0.9.tbd 등이 libc++로 변경된 이유라 함
1) xcode9에서 해당 파일을 찾아 xcode10 에 복사
[Konlpy 설치] macOS에서 konlpy설치 시 오류
파이썬에서 NLP를 다룰 때 한 줄기 빛 같은 존재인 Konlpy를 설치할 때, 사전 필요 library인 JPype1를 설치하는 도중
medium.com
[ 해결 방법]
(tensorflow) iMac-Pro-mlsee21:colorcorrect-0.8 mslee21$ export MACOSX_DEPLOYMENT_TARGET=10.10
(tensorflow) iMac-Pro-mlsee21:colorcorrect-0.8 mslee21$ CFLAGS='-stdlib=libc++' pip install colorcorrect
Collecting colorcorrect
Requirement already satisfied: numpy in /Users/mslee21/anaconda3/envs/tensorflow/lib/python3.6/site-packages (from colorcorrect) (1.15.4)
Requirement already satisfied: Pillow in /Users/mslee21/anaconda3/envs/tensorflow/lib/python3.6/site-packages (from colorcorrect) (5.4.1)
Requirement already satisfied: six in /Users/mslee21/anaconda3/envs/tensorflow/lib/python3.6/site-packages (from colorcorrect) (1.12.0)
Building wheels for collected packages: colorcorrect
Running setup.py bdist_wheel for colorcorrect ... done
Stored in directory: /Users/mslee21/Library/Caches/pip/wheels/83/ac/d2/c18fefea61cd3e13ff9247690d8cfbdce0ef74e2ed0bc76b20
Successfully built colorcorrect
Installing collected packages: colorcorrect
Successfully installed colorcorrect-0.8
### 전체적 설치 참고: https://tech.amikelive.com/node-669/guide-installing-cuda-toolkit-9-1-on-ubuntu-16-04/
Guide: Installing Cuda Toolkit 9.1 on Ubuntu 16.04
With advances in GPU technologies, performing complex computation is not an exclusive feat of multicore CPUs anymore. It is not uncommon to perform computation for linear algebra, image and video p…
tech.amikelive.com
1. apt 패키지 업데이트
euler@euler:~$ sudo apt-get update
[sudo] password for euler:
Hit:1 http://kr.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Fetched 109 kB in 2s (54.1 kB/s)
Reading package lists... Done
euler@euler:~$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
libllvm5.0 snapd-login-service
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[Error 1]
E: Could not get lock /var/lib/dpkg/lock - open (11 Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/) is another process using it?
[solution] (참고: https://www.hahwul.com/2016/08/debian-apt-get-could-not-get-lock.html )
#> rm /var/lib/dpkg/lock
2. GPU가 CUDA를 지원하는지 확인 ( 치포스 750Ti, 1080Ti)
euler@euler:~$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
3. Ubuntu 버전 확인
euler@euler:~$ uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
4. GCC 버전확인
euler@euler:~$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
5. kernel 버전 확인
euler@euler:~$ uname -r
4.13.0-36-generic
6. kernel header 설치
euler@euler:~$ sudo apt-get install linux-headers-$(uname -r)
Reading package lists... Done
Building dependency tree
Reading state information... Done
linux-headers-4.13.0-36-generic is already the newest version (4.13.0-36.40~16.04.1).
The following packages were automatically installed and are no longer required:
libllvm5.0 snapd-login-service
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
7. Cuda 다운로드
https://developer.nvidia.com/cuda-downloads에서 deb (network) 버전을 다운로드함
8. Cuda 툴킷 설치
euler@euler:~/Downloads$ sudo dpkg -i cuda-repo-ubuntu1604_10.1.105-1_amd64.deb Selecting previously unselected package cuda-repo-ubuntu1604.
(Reading database ... 214063 files and directories currently installed.)
Preparing to unpack cuda-repo-ubuntu1604_10.1.105-1_amd64.deb ...
Unpacking cuda-repo-ubuntu1604 (10.1.105-1) ...
Setting up cuda-repo-ubuntu1604 (10.1.105-1) ...
The public CUDA GPG key does not appear to be installed.
To install the key, run this command:
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
설치하는 중에 CUDA GPG 키를 설치하라고 함. 나온대로 설치를 진행
euler@euler:~/Downloads$ sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
Executing: /tmp/tmp.fnXOs1IEOM/gpg.1.sh --fetch-keys
http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
gpg: key 7FA2AF80: public key "cudatools <cudatools@nvidia.com>" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
apt 패키지 인덱스를 다시 업데이트
euler@euler:~/Downloads$ sudo apt-get update
Hit:1 http://kr.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:4 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 InRelease
Get:5 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release [564 B]
Get:6 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release.gpg [801 B]
Get:7 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Packages [209 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Fetched 320 kB in 1s (216 kB/s)
Reading package lists... Done
Cuda 설치
euler@euler:~/Downloads$ sudo apt-get install cuda
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libllvm5.0 snapd-login-service
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
bbswitch-dkms ca-certificates-java cuda-10-1 cuda-command-line-tools-10-1
cuda-compiler-10-1 cuda-cudart-10-1 cuda-cudart-dev-10-1 cuda-cufft-10-1
cuda-cufft-dev-10-1 cuda-cuobjdump-10-1 cuda-cupti-10-1 cuda-curand-10-1
cuda-curand-dev-10-1 cuda-cusolver-10-1 cuda-cusolver-dev-10-1
...............................
시스템 reboot
nvidia 그래픽드라이버 설치확인
(참고: https://hiseon.me/2018/02/17/install_nvidia_driver/)
euler@euler:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 418.67 Sat Apr 6 03:07:24 CDT 2019
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)
euler@euler:~$ lspci -k
....
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750 Ti]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_418_drm, nvidia_418
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device 36e2
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device 3752
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_418_drm, nvidia_418
02:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device 3752
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
.....
드라이버 버전 확인
euler@euler:~$ sudo cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 418.67 Sat Apr 6 03:07:24 CDT 2019
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)
euler@euler:~$ nvidia-smi
Tue Jun 11 20:45:23 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 750 Ti Off | 00000000:01:00.0 Off | N/A |
| 45% 37C P8 1W / 46W | 1MiB / 981MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:02:00.0 On | N/A |
| 0% 54C P2 58W / 250W | 350MiB / 11177MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 1 1062 G /usr/lib/xorg/Xorg 184MiB |
| 1 1839 G compiz 163MiB |
+-----------------------------------------------------------------------------+
CUDA 경로 설정
euler@euler:~$ nano ~/.profile
PATH="/usr/local/cuda-10.1/bin:$HOME/bin:$HOME/.local/bin:$PATH"
NVIDIA persistence service 설정
( 참고사이트와 달리 /usr/lib/... 가 아니라 /lib/ 에 있음)
euler@euler:~$ cd /lib/systemd/system && sudo touch nvidia-persistenced
euler@euler:/lib/systemd/system$ sudo nano nvidia-persistenced
[Unit]
Description=NVIDIA Persistence Daemon
Wants=syslog.target
[Service]
Type=forking
PIDFile=/var/run/nvidia-persistenced/nvidia-persistenced.pid
Restart=always
ExecStart=/usr/bin/nvidia-persistenced --verbose
ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced
[Install]
WantedBy=multi-user.target
persistence 실행
euler@euler:/lib/systemd/system$ sudo systemctl enable nvidia-persistenced
euler@euler:/lib/udev/rules.d$ sudo nano 40-vm-hotadd.rules
.....
# Memory hotadd request
SUBSYSTEM=="memory", ACTION=="add", DEVPATH=="/devices/system/memory/memory[0-9]*", TEST=="state", ATTR{state}="online"
......
시스템 reboot
euler@euler:~$ sudo reboot
Nvidia driver버전 확인
euler@euler:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 418.67 Sat Apr 6 03:07:24 CDT 2019
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)
nvidia CUDA toolkit 버전 확인
euler@euler:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:10:27_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168
(이상한 점)
CUDA toolkit 을 설치중 아래와 같은 에러 발생
----
Errors were encountered while processing:
/var/cache/apt/archives/nvidia-cuda-dev_7.5.18-0ubuntu1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
----
일단 확인하기는 아래와 같이 해서 재설치
mslee21@euler:~/Downloads$ sudo dpkg --configure -a
[sudo] password for mslee21:
dpkg: dependency problems prevent configuration of nvidia-cuda-toolkit:
nvidia-cuda-toolkit depends on nvidia-cuda-dev (= 7.5.18-0ubuntu1); however:
Package nvidia-cuda-dev is not installed.
dpkg: error processing package nvidia-cuda-toolkit (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
nvidia-cuda-toolkit
mslee21@euler:~/Downloads$ sudo apt-get clean
