Prag-o-matic

A blog.

cx_Oracle with AWS Lambda

This work was part of a demonstration to a development team regarding the use of AWS for web-based applications. I was demonstrating a few approaches:

  1. Using a Lift & Shift approach that replicated the Oracle Database + PHP web app environment:

    1. Input data (CSV) sent to S3 via a nightly upload

    2. Change to the S3 bucket triggers a Lambda to update the RDS-based Database

    3. PHP application largely the same but with alterations that made it more self-contained

  2. Going "serverless" by:

    1. Input data (CSV) sent to S3 via a nightly upload

    2. Change to the S3 bucket triggers a Lambda that performs an ETL process to generate a set of static JSON API files to be stored in another S3 bucket

    3. An API Gateway then maps an API to the underlying S3-based JSON API data

    4. Two web apps were then developed by the dev team: a PHP that used the API and a JavaScript-based app that ran client-side.

In order to achieve item 1.b we needed to setup a Python 2 Lambda that can connect to the Oracle database. Support for this is via the cx_Oracle module but it requires more than a pip install as the module expects access to the Oracle client. You can see what happens by running up a container (docker run --rm -ti python:3 /bin/bash) and trying out the module:

root@6bba52c06926:/# python -m pip install cx_Oracle --pre
Collecting cx_Oracle
  Downloading cx_Oracle-6.0rc1-cp36-cp36m-manylinux1_x86_64.whl (487kB)
    100% |████████████████████████████████| 491kB 2.0MB/s
Installing collected packages: cx-Oracle
Successfully installed cx-Oracle-6.0rc1
root@6bba52c06926:/# python
Python 3.6.1 (default, May 11 2017, 22:14:44)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cx_Oracle.DatabaseError: DPI-1047: Oracle Client library cannot be loaded: libclntsh.so: cannot open shared object file: No such file or directory. See https://oracle.github.io/odpi/doc/installation.html for help

I have a few images with cx_Oracle running but needed to package up a distributable that can go to Lambda. As you’ll see in the Bitbucket repo’s Readme, I ended up with the following process:

  1. Create a container image (based on amazonlinux) that:

    1. Provides the requisite software and configuration

    2. Packages up the required shared libraries (after a lot of web searching and testing)

  2. Run up an instance of the image and copy out the package

  3. Create a Lambda with:

    1. The package as the Lambda source

    2. The correct envrionment variables for accessing the shared libraries

The end product is a repeatable deployment process - we just needed to provide the Python script to read from S3 into the Oracle RDS DB.