[Python][Solution] Install awswrangler Package on ARM (AArch64)

Enviornment : Amazon Linux 2 / Python 3.7.*

 

 

Problem

AWS Gravtion Instance (ARM 프로세서) 에서 awswrangler package 설치 시 에러 발생

[ec2-user@ip-10-241-104-135 ~]$ pip install awswrangler
Defaulting to user installation because normal site-packages is not writeable
Collecting awswrangler
  Downloading awswrangler-2.16.1-py3-none-any.whl (248 kB)
     |????????????????????????????????| 248 kB 13.3 MB/s 
...
Collecting multidict<7.0,>=4.5
  Downloading multidict-6.0.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (96 kB)
     |????????????????????????????????| 96 kB 14.1 MB/s 
Collecting async-timeout<5.0,>=4.0.0a3
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Installing collected packages: backoff, idna, charset-normalizer, certifi, requests, requests-aws4auth, et-xmlfile, openpyxl, asn1crypto, scramp, zipp, typing-extensions, importlib-metadata, pg8000, pyparsing, packaging, redshift-connector, python-utils, progressbar2, decorator, ply, jsonpath-ng, numpy, pyarrow, pymysql, aenum, isodate, nest-asyncio, asynctest, multidict, yarl, frozenlist, aiosignal, attrs, async-timeout, aiohttp, gremlinpython, opensearch-py, awswrangler, soupsieve

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

python37-sagemaker-pyspark 1.4.1 requires pyspark==2.3.4, which is not installed.
redshift-connector 2.0.908 requires requests<2.28.1,>=2.23.0, but you'll have requests 2.28.1 which is incompatible.

 

 

 

Solution

pip install git+https://github.com/awslabs/aws-data-wrangler.git@main

 

AWS Gravtion Instance ( ARM ISA 기반 프로세서) 의 경우 기존에 Intel x86과 컴파일 방식이 다르기 때문에 사용하는 언어/패키지 등에 따라 프로세서별로 다르게 동작할 수 있다.

Python의 경우 Compile 언어가 아니지만 사용하는 패키지는 C/C++ 기반으로 개발되어 Compile 과정이 필요한 경우가 있다. 

 

PyArrow 가 4.0.0 버전부터 ARM 프로세서를 지원하게 되었고, awswrangler 2.8.0 부터 패치가 된 상태인데 현재 그냥 pip install awswrangler 명령어로 설치 시 2.15 버전으로 설치하려고 하여 패키지 설치 오류가 발생한 것으로 추정된다. 

 

때문에 Github을 통해 최신 버전의 awswrangler를 설치할 수 있도록 했고 문제 없이 사용이 가능함을 확인했다.

 

 

 

 

 

Reference
https://github.com/awslabs/aws-data-wrangler/issues/557
https://github.com/awslabs/aws-data-wrangler/releases/tag/2.8.0