Skip to main content

Writing Python

Use Python to:

  • Transform the output of previous Steps by referencing their output (ex. Step1.output)
  • Write logic referencing UI components (ex. Table1.selectedRow.id)
  • Return data to be consumed by subsequent steps and UI components

Supported Libraries

When using Superblocks Cloud or the latest version of the on-premise agent, use any of the following Python libraries in Superblocks Applications, Workflows, or Scheduled Jobs. The latest list of libraries can also be found in the GitHub repo here.

To request a new library, email help@superblockshq.com with the library name and version.

All Libraries

aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
arrow==1.1.1
asn1crypto==1.5.1
asttokens==2.0.5
async-timeout==4.0.2
attrs==21.4.0
Authlib==1.0.1
backcall==0.2.0
beautifulsoup4==4.11.1
boto==2.49.0
boto3==1.22.10
botocore==1.25.10
Brotli==1.0.9
bs4==0.0.1
cachetools==5.0.0
canalyst-candas==0.0.35
certifi==2021.5.30
cffi==1.15.0
charset-normalizer==2.0.4
click==8.1.3
ConfigArgParse==1.5.3
cryptography==36.0.2
cssselect==1.1.0
cvxpy==1.1.15
cycler==0.11.0
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.13
ecos==2.0.10
et-xmlfile==1.1.0
executing==0.8.3
fake-useragent==0.1.11
feedparser==6.0.8
Flask==2.2.2
Flask-BasicAuth==0.2.0
Flask-Cors==3.0.10
fonttools==4.33.3
fred==3.1
fredapi==0.5.0
frozenlist==1.3.0
future==0.18.2
gevent==21.12.0
geventhttpclient==2.0
gocardless-pro==1.32.0
google-api-core==2.7.3
google-auth==2.6.6
google-auth-oauthlib==0.5.3
google-cloud-bigquery==2.31.0
google-cloud-core==2.3.0
google-cloud-storage==1.43.0
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.56.0
graphviz==0.20
greenlet==1.1.2
grpcio==1.46.0
grpcio-status==1.46.0
gspread==5.5.0
idna==3.2
importlib-metadata==4.11.3
ipython==8.3.0
itsdangerous==2.1.2
jedi==0.18.1
jeepney==0.8.0
Jinja2==3.1.2
jira==3.2.0
jmespath==1.0.0
joblib==1.1.0
jsondiff==2.0.0
jsonpickle==2.1.0
jycm==1.3.0
kaleido==0.2.1
keyring==23.6.0
kiwisolver==1.4.2
locust==2.10.2
lxml==4.9.1
MarkupSafe==2.1.1
matplotlib==3.5.2
matplotlib-inline==0.1.3
moncli==2.0.14
msgpack==1.0.4
multidict==6.0.2
mysql-connector-python==8.0.29
networkx==2.8
numexpr==2.8.1
numpy==1.22.1
oauthlib==3.2.0
openpyxl==3.0.9
oscrypto==1.3.0
osqp==0.6.2.post5
packaging==21.3
pandas==1.3.5
parse==1.19.0
parso==0.8.3
patsy==0.5.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.1.1
plotly==5.5.0
prompt-toolkit==3.0.29
proto-plus==1.20.3
protobuf==3.20.1
psutil==5.9.1
psycopg2==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyaes==1.6.1
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycountry==22.3.5
pycparser==2.21
pycryptodomex==3.14.1
pydantic==1.9.0
pydot==1.4.2
pyee==8.2.2
Pygments==2.12.0
PyJWT==2.4.0
pyOpenSSL==22.0.0
pyparsing==3.0.8
pyppeteer==1.0.2
pyquery==1.4.3
python-box==5.4.1
python-dateutil==2.8.2
python-docx==0.8.11
python-graphql-client==0.4.3
pytz==2021.1
pyvis==0.2.1
PyYAML==6.0
pyzmq==22.3.0
qdldl==0.1.5.post2
quickchart.io==1.0.1
requests==2.26.0
requests-futures==1.0.0
requests-html==0.10.0
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
roundrobin==0.0.2
rsa==4.8
s3transfer==0.5.2
schematics==2.1.1
scikit-learn==1.0.1
scipy==1.7.3
scs==3.2.0
SecretStorage==3.3.2
sgmllib3k==1.0.0
simple-salesforce==1.11.4
simplejson==3.17.6
six==1.16.0
slack-sdk==3.15.1
snowflake-connector-python==2.7.8
soupsieve==2.3.2.post1
stack-data==0.2.0
statsmodels==0.13.2
stripe==3.4.0
tablib==3.2.1
Telethon==1.24.0
tenacity==8.0.1
threadpoolctl==3.1.0
tqdm==4.64.0
traitlets==5.2.0
typing_extensions==4.2.0
urllib3==1.26.6
w3lib==1.22.0
wcwidth==0.2.5
websockets==10.3
Werkzeug==2.2.2
wrapt==1.14.1
xero-python==1.18.0
yahoo-fin==0.8.9.1
yarl==1.7.2
zipp==3.8.0
zope.event==4.5.0
zope.interface==5.4.0

Referencing UI components and previous steps

Superblocks converts all UI and previous steps into Python namespaces, which lets you use dot notation instead of array notation to access properties. For example, Table1.selectedRow.name. Note, you can still access the __dict__ of the namespace if your code requires a dictionary as input.

Autocomplete is provided within python and javascript steps

info

Press CMD + K to reference available Superblocks variables from Python.

Examples

HTTP Requests

Make an HTTP request using the requests library:

import requests
import urllib.parse

key = ''
address_encoded = urrlib.parse.quote_plus('')
return requests.get(f'https://maps.googleapis.com/maps/api/geocode/json?address={address_encoded}&key={key}').json()

The requests library is easy to use and solves most use cases for making simple HTTP requests in Python. However, when making a large number of HTTP requests where performance is a priority, speed can be optimized by using the asyncio and aiohttp libraries to run Python requests asynchronously.

Here is an example that gets all of the users from the paginated Zendesk Users API:

import aiohttp
import asyncio
import requests
import math

#Zendesk config
email = '<EMAIL>'
api_token = '<TOKEN>'
url = 'https://<DOMAIN>.zendesk.com/api/v2/users'

#Initial request to get page count
resp = requests.get(url, auth=(f'{email}/token', api_token))
resp_json = resp.json()
page_count = resp_json.get("count")
records_per_page = 100

async def get_users(session, url):
async with session.get(url, auth=aiohttp.BasicAuth(f'{email}/token', api_token)) as resp:
resp_json = await resp.json()
user_list = resp_json['users']
return user_list

async def main():
async with aiohttp.ClientSession() as session:
tasks = []
for number in range(1, math.ceil(page_count/records_per_page)):
per_page_url = url + f'?page={number}'
tasks.append(asyncio.ensure_future(get_users(session, per_page_url)))

users = await asyncio.gather(*tasks)

#Combine individual lists
all_users = []
for user_list in users:
all_users+=user_list

return all_users

return asyncio.run(main())
info

When calling any third party REST APIs, be aware of API rate limits. The example above issues total_zendesk_users/100 requests to Zendesk.

While the code is more verbose, it results in a large performance gain. Fetching all 10,000+ users now takes less than 2 seconds versus more than 50 seconds when getting the same data synchronously.

http request with python requests
http request with python asyncio and aiohttp libraries

For more information on using these libraries, this blog is a helpful resource.

Machine Learning

Create a pandas DataFrame from a previous step, and return the serialized DataFrame:

import json
import pandas as pd

df = pd.DataFrame.from_dict(Step1.output)
return json.loads(df.to_json(orient='records'))

Convert a numpy ndarray to JSON:

import numpy as np

arr = np.array([[ 1, 2, 3],
[ 4, 2, 5]])

return arr.tolist()

Parse CSV

Parse CSV uploaded with FilePicker to display contents in a table:

from io import StringIO
import csv
import json

scsv = FilePicker1.files[0].readContents()
f = StringIO(scsv)

reader = csv.DictReader(f, delimiter=',')

return [row for row in reader]

Graphing

Create advanced charts with Plotly:

import plotly.express as px
import numpy as np
df = px.data.gapminder().query("year == 2007")
fig = px.sunburst(df, path=['continent', 'country'], values='pop',
color='lifeExp', hover_data=['iso_alpha'],
color_continuous_scale='RdBu',
color_continuous_midpoint=np.average(df['lifeExp'], weights=df['pop']))
return fig

Life expectancy sunburst chart with plotly

For more information about creating charts with Plotly in Superblocks, see our charts docs.

Connect to AWS services with Boto3

For AWS services where Superblocks doesn't yet have a native integration, or use cases where you need more control or custom logic beyond the integration, you can connect to any AWS service in a Python step that uses the Boto3 library.

For example, create a workflow that fetches secrets from AWS Secrets Manager. Or create an application that lists all files within a subfolder in an S3 bucket:

import boto3

s3 = boto3.client('s3', aws_access_key_id = '<YOUR_ACCESS_KEY_ID>', aws_secret_access_key = '<YOUR_SECRET_KEY>')

files = []
results = s3.list_objects_v2(Bucket='<BUCKET_NAME>', Prefix='<PATH_TO_FOLDER>')
for file in results.get('Contents'):
if file['Size']!=0: #exclude the folder itself
file['LastModified'] = file['LastModified'].strftime("%Y-%m-%dT%H:%M:%S.%fZ") #stringify the datetime
files.append(file)

return files

You can find more code examples in the Boto3 docs here.

Python in Applications, Workflows and Scheduled Jobs

Application

Here we have an API that gets details about orders from a Postgres database, and shipping status for those orders from a Snowflake data warehouse. The final step uses Python to merge the data from Postgres and Snowflake. We can then display the combined data in our applicaton UI in the form of a grid.

Python can be used to merge data from different sources and pass back to the UI for display within components

Workflow

The application above also includes a button for refunding an order. We can extract the refund logic into a workflow to be used by other Superblocks applications, internal applications, or webhooks.

Make use of event triggers on UI components to call Superblocks workflows.

This particular Refund_Order workflow includes a Python step that calls our shipping vendor’s API to generate a shipping label and attach a return merchandise authorization (RMA) number to it.

Python code can be used within workflows to cater for custom logic

Then we pass this information to an internal GraphQL API that processes the refund, and finally send a confirmation email with tracking information to the customer.

For more detailed information on creating this workflow, check out the Refund Order with Python, GraphQL, and Email in the workflow library.

Scheduled Job

Based on the previous refund activity, we also want to regularly update our inventory prediction model. We can do this with a scheduled job that runs every weekday at 8pm. It uses the orders and refunds submitted over the last day to train a linear model with sklearn in Python, serializes the model, and uploads it to S3 for the data science team to use.

Scikit learn module is supported within python steps allowing for the use of machine learning

Troubleshooting

Returning JSON

The return value of your Python code must be JSON-serializable. Superblocks does not serialize complex types like what pandas and numpy use. The error message Error: 'numpy.ndarray' object has no attribute '__dict__' indicates a serialization error. To solve this error you should call the proper functions, for example .tolist().

Note that map() in Python 3 returns an iterator and will result in an error if returned.

Python errors are shown with context to ease troubleshooting

Casting the map to a list is required before returning.

Output from API is a list