Writing Python
The supported Python version is 3.10.4
Use Python to:
- Transform the output of previous Steps by referencing their output (ex.
Step1.output) - Write logic referencing UI components (ex.
Table1.selectedRow.id) - Return data to be consumed by subsequent steps and UI components
Supported Libraries
When using Superblocks Cloud or the latest version of the On-Premise Agent, import any of the following Python libraries in Superblocks Applications, Workflows, or Scheduled Jobs. Note, some of these libraries are transitive dependencies of other libraries included in the Superblocks Python environment, so they may not be useful in Superblocks on their own.
All Libraries
Acquisition==4.13
aiohttp==3.9.4
Acquisition==4.13
aiohttp==3.9.5
aiosignal==1.3.1
appdirs==1.4.4
appnope==0.1.4
arrow==1.3.0
asana==2.0.0
asn1crypto==1.5.1
assemblyai==0.26.0
asttokens==2.4.1
async-timeout==4.0.3
asyncio==3.4.3
attrs==22.2.0
AuthEncoding==4.3
Authlib==1.3.1
backcall==0.2.0
backoff==2.2.1
beautifulsoup4==4.12.3
bidict==0.23.1
boto==2.49.0
boto3==1.22.10
botocore==1.25.10
Brotli==1.1.0
bs4==0.0.2
BTrees==4.11.3
CacheControl==0.14.0
cachetools==5.3.3
certifi==2024.7.4
cffi==1.15.0
charset-normalizer==2.0.12
click==8.1.7
cmake==3.24.1.1
ConfigArgParse==1.7
croniter==2.0.7
cryptography==36.0.2
cssselect==1.1.0
cvxpy==1.1.24
cycler==0.11.0
Cython==0.29.37
dataclasses-json==0.5.7
DateTime==4.9
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.14
distro==1.9.0
dnspython==2.6.1
ecos==2.0.13
et-xmlfile==1.1.0
exceptiongroup==1.2.1
executing==0.8.3
ExtensionClass==4.9
fake-useragent==0.1.14
feedparser==6.0.11
filelock==3.15.4
firebase-admin==6.5.0
Flask==2.2.5
Flask-BasicAuth==0.2.0
Flask-Cors==3.0.10
fonttools==4.43.0
fred==3.1
fredapi==0.5.2
frozendict==2.3.7
frozenlist==1.3.1
fsspec==2024.3.1
future==0.18.3
geojson==2.5.0
gevent==21.12.0
geventhttpclient==2.0.12
gocardless-pro==1.32.0
google-api-core==2.18.0
google-api-python-client==2.95.0
google-auth==2.29.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.3
google-cloud-bigquery==2.31.0
google-cloud-core==2.3.0
google-cloud-firestore==2.16.0
google-cloud-functions==1.16.3
google-cloud-pubsub==2.13.0
google-cloud-storage==1.43.0
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.63.0
graphviz==0.20.3
greenlet==1.1.3.post0
grpc-google-iam-v1==0.12.4
grpcio==1.53.2
grpcio-status==1.46.0
gspread==5.5.0
h11==0.14.0
html5lib==1.1
httpcore==1.0.5
httplib2==0.22.0
httpx==0.27.0
huggingface-hub==0.23.0
idna==3.7
importlib-metadata==4.11.3
ipython==8.10.0
isodate==0.6.1
itsdangerous==2.1.2
jedi==0.18.1
jeepney==0.8.0
Jinja2==3.1.4
jira==3.2.0
jmespath==1.0.1
joblib==1.3.2
jsondiff==2.0.0
jsonpickle==2.1.0
jycm==1.3.0
kaleido==0.2.1
keyring==23.6.0
kiwisolver==1.4.5
langchain==0.0.200
langchainplus-sdk==0.0.16
locust==2.10.2
loguru==0.6.0
lxml==4.9.4
MarkupSafe==2.1.5
marshmallow==3.18.0
marshmallow-enum==1.5.1
matplotlib==3.5.2
matplotlib-inline==0.1.6
mccabe==0.7.0
moncli==2.0.14
msgpack==1.0.8
multidict==6.0.5
multipart==0.2.4
multitasking==0.0.11
mypy_extensions==0.4.4
mysql-connector-python==8.0.29
nest-asyncio==1.5.6
networkx==2.8.8
nltk==3.8.1
numexpr==2.8.4
numpy==1.26.4
oauthlib==3.2.2
openai==0.27.2
openapi-schema-pydantic==1.2.4
openpyxl==3.0.9
opentelemetry-api==1.15.0
opentelemetry-exporter-otlp-proto-http==1.15.0
opentelemetry-proto==1.15.0
opentelemetry-sdk==1.15.0
opentelemetry-semantic-conventions==0.36b0
oscrypto==1.3.0
osqp==0.6.7
packaging==21.3
pandas==1.5.3
parse==1.19.0
parso==0.8.3
pathspec==0.10.1
patsy==0.5.6
peewee==3.17.1
Persistence==3.6
persistent==4.9.3
pexpect==4.8.0
phonenumberslite==8.13.31
pickleshare==0.7.5
pillow==10.3.0
pinecone-client==2.2.4
platformdirs==4.2.2
plotly==5.19.0
postgresql-audit==0.17.1
prometheus-client==0.14.1
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==3.20.3
psutil==5.9.8
psycopg2==2.9.9
ptyprocess==0.7.0
pure-eval==0.2.2
pyaes==1.6.1
pyarrow==14.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.10.0
pycodestyle==2.9.1
pycountry==22.3.5
pycparser==2.21
pycryptodomex==3.19.1
pydantic==1.10.14
pydash==6.0.2
pydot==1.4.2
pyee==8.2.2
Pygments==2.17.2
PyJWT==2.8.0
pymongo==4.6.3
pymongo-auth-aws==1.1.0
pymsteams==0.2.2
pyodbc==5.0.1
pyOpenSSL==22.0.0
pyowm==3.3.0
pyparsing==3.0.8
pypdf==3.17.4
pypng==0.20220715.0
pyppeteer==1.0.2
pyquery==1.4.3
PySocks==1.7.1
python-barcode==0.15.1
python-box==5.4.1
python-dateutil==2.8.2
python-docx==0.8.11
python-dotenv==0.21.1
python-engineio==4.8.2
python-gettext==4.1
python-graphql-client==0.4.3
python-json-logger==2.0.7
python-socketio==5.7.2
pytz==2023.3
pyvis==0.2.1
PyYAML==6.0.1
pyzipper==0.3.6
pyzmq==22.3.0
qdldl==0.1.7.post0
qrcode==7.4.2
quickchart.io==1.0.1
redis==4.6.0
regex==2023.12.25
requests==2.32.0
requests-file==1.5.1
requests-futures==1.0.1
requests-html==0.10.0
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
roundrobin==0.0.4
rsa==4.8
s3transfer==0.5.2
schematics==2.1.1
scikit-learn==1.5.0
scipy==1.14.0
scs==3.2.0
SecretStorage==3.3.3
sgmllib3k==1.0.0
simple-salesforce==1.11.4
simple-websocket==1.0.0
simplejson==3.17.6
six==1.16.0
SJSON==2.0.3
slack-sdk==3.15.1
sniffio==1.3.1
snowflake-connector-python==3.10.1
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.52
SQLAlchemy-Utils==0.41.2
stack-data==0.2.0
statsmodels==0.13.2
stripe==3.5.0
tablib==3.2.1
tabulate==0.9.0
Telethon==1.34.0
tenacity==8.1.0
threadpoolctl==3.1.0
thrift==0.16.0
tldextract==5.1.2
tokenizers==0.19.1
tomli==2.0.1
tomlkit==0.12.5
tqdm==4.66.3
traitlets==5.2.0
transaction==3.0.1
types-python-dateutil==2.9.0.20240316
types-requests==2.28.11.2
types-simplejson==3.17.7
types-urllib3==1.26.25.14
typing-inspect==0.8.0
typing_extensions==4.11.0
ujson==5.10.0
uritemplate==4.1.1
urllib3==1.26.19
w3lib==1.22.0
wcwidth==0.2.13
webencodings==0.5.1
websockets==10.3
Werkzeug==3.0.3
wrapt==1.14.1
wsproto==1.2.0
xero_python==1.18.0
yahoo-fin==0.8.9.1
yarl==1.8.1
yfinance==0.2.37
zeep==4.2.1
zExceptions==4.2
zipp==3.19.1
zope.browser==2.4
zope.component==5.0.1
zope.configuration==4.4.1
zope.contenttype==4.6
zope.deferredimport==4.4
zope.deprecation==4.4.0
zope.event==4.5.0
zope.exceptions==4.5
zope.hookable==5.2
zope.i18n==4.9.0
zope.i18nmessageid==5.0.1
zope.interface==5.4.0
zope.location==4.2
zope.proxy==4.5.1
zope.publisher==6.1.0
zope.schema==6.2.1
zope.security==5.4
zope.testing==4.10
Note, if you're using the On-Premise Agent, it's also possible to import custom packages by creating a custom Docker image.
Referencing UI components and previous steps
Superblocks converts all UI and previous steps into Python namespaces, which lets you use dot notation instead of array notation to access properties. For example, Table1.selectedRow.name. Note, you can still access the __dict__ of the namespace if your code requires a dictionary as input.
Press CMD + K to reference available Superblocks variables from Python.
Examples
HTTP Requests
Make an HTTP request using the requests library:
import requests
import urllib.parse
key = ''
address_encoded = urrlib.parse.quote_plus('')
return requests.get(f'https://maps.googleapis.com/maps/api/geocode/json?address={address_encoded}&key={key}').json()
The requests library is easy to use and solves most use cases for making simple HTTP requests in Python. However, when making a large number of HTTP requests where performance is a priority, speed can be optimized by using the asyncio and aiohttp libraries to run Python requests asynchronously.
Here is an example that gets all of the users from the paginated Zendesk Users API:
import aiohttp
import asyncio
import requests
import math
#Zendesk config
email = '<EMAIL>'
api_token = '<TOKEN>'
url = 'https://<DOMAIN>.zendesk.com/api/v2/users'
#Initial request to get page count
resp = requests.get(url, auth=(f'{email}/token', api_token))
resp_json = resp.json()
page_count = resp_json.get("count")
records_per_page = 100
async def get_users(session, url):
async with session.get(url, auth=aiohttp.BasicAuth(f'{email}/token', api_token)) as resp:
resp_json = await resp.json()
user_list = resp_json['users']
return user_list
async def main():
async with aiohttp.ClientSession() as session:
tasks = []
for number in range(1, math.ceil(page_count/records_per_page)):
per_page_url = url + f'?page={number}'
tasks.append(asyncio.ensure_future(get_users(session, per_page_url)))
users = await asyncio.gather(*tasks)
#Combine individual lists
all_users = []
for user_list in users:
all_users+=user_list
return all_users
return asyncio.run(main())
When calling any third party REST APIs, be aware of API rate limits. The example above issues total_zendesk_users/100 requests to Zendesk.
While the code is more verbose, it results in a large performance gain. Fetching all 10,000+ users now takes less than 2 seconds versus more than 50 seconds when getting the same data synchronously.
For more information on using these libraries, this blog is a helpful resource.
Machine Learning
Create a pandas DataFrame from a previous step, and return the serialized DataFrame:
import json
import pandas as pd
df = pd.DataFrame.from_dict(Step1.output)
return json.loads(df.to_json(orient='records'))
Convert a numpy ndarray to JSON:
import numpy as np
arr = np.array([[ 1, 2, 3],
[ 4, 2, 5]])
return arr.tolist()
Parse CSV
Parse CSV uploaded with FilePicker to display contents in a table:
from io import StringIO
import csv
import json
scsv = FilePicker1.files[0].readContents()
f = StringIO(scsv)
reader = csv.DictReader(f, delimiter=',')
return [row for row in reader]
Graphing
Create advanced charts with Plotly:
import plotly.express as px
import numpy as np
df = px.data.gapminder().query("year == 2007")
fig = px.sunburst(df, path=['continent', 'country'], values='pop',
color='lifeExp', hover_data=['iso_alpha'],
color_continuous_scale='RdBu',
color_continuous_midpoint=np.average(df['lifeExp'], weights=df['pop']))
return fig
For more information about creating charts with Plotly in Superblocks, see our charts docs.
Connect to AWS services with Boto3
For AWS services where Superblocks doesn't yet have a native integration, or use cases where you need more control or custom logic beyond the integration, you can connect to any AWS service in a Python step that uses the Boto3 library.
For example, create a workflow that fetches secrets from AWS Secrets Manager. Or create an application that lists all files within a subfolder in an S3 bucket:
import boto3
s3 = boto3.client('s3', aws_access_key_id = '<YOUR_ACCESS_KEY_ID>', aws_secret_access_key = '<YOUR_SECRET_KEY>')
files = []
results = s3.list_objects_v2(Bucket='<BUCKET_NAME>', Prefix='<PATH_TO_FOLDER>')
for file in results.get('Contents'):
if file['Size']!=0: #exclude the folder itself
file['LastModified'] = file['LastModified'].strftime("%Y-%m-%dT%H:%M:%S.%fZ") #stringify the datetime
files.append(file)
return files
You can find more code examples in the Boto3 docs here.
Python in Applications, Workflows and Scheduled Jobs
Application
Here we have an API that gets details about orders from a Postgres database, and shipping status for those orders from a Snowflake data warehouse. The final step uses Python to merge the data from Postgres and Snowflake. We can then display the combined data in our applicaton UI in the form of a grid.
Workflow
The application above also includes a button for refunding an order. We can extract the refund logic into a workflow to be used by other Superblocks applications, internal applications, or webhooks.
This particular Refund_Order workflow includes a Python step that calls our shipping vendor’s API to generate a shipping label and attach a return merchandise authorization (RMA) number to it.
Then we pass this information to an internal GraphQL API that processes the refund, and finally send a confirmation email with tracking information to the customer.
For more detailed information on creating this workflow, check out the Refund Order with Python, GraphQL, and Email in the workflow library.
Scheduled Job
Based on the previous refund activity, we also want to regularly
update our inventory prediction model. We can do this with a scheduled job that runs every weekday at 8pm. It uses the orders and refunds submitted over the last day to train a linear model with sklearn in Python, serializes the model, and uploads it to S3 for the data science team to use.
Troubleshooting
Returning JSON
The return value of your Python code must be JSON-serializable. Superblocks does not serialize complex types like what pandas and numpy use. The error message Error: 'numpy.ndarray' object has no attribute '__dict__' indicates a serialization error. To solve this error you should call the proper functions, for example .tolist().
Note that map() in Python 3 returns an iterator and will result in an error if returned.
Casting the map to a list is required before returning.
