Pythonic Analysis Agent

Pythonic conversational analytics driven by LLMs.
DataAnalysis’s Pythonic analysis offers users a complete toolkit for performing machine learning-based analysis, utilizing Python’s broad selection of libraries like NumPy, pandas, scikit-learn, and statsmodels. Users can harness Python’s extensive ecosystem of machine learning tools and methods to extract actionable insights from their data and address complex analytical challenges.

Getting Started

DataAnalysis includes four simple steps:

1. Install aih
   Install the aih package with the data-analysis variant.

2. Create an instance
   Instantiate the DataAnalysis class with your chosen analysis type and API key.

3. Load data
   Use the get_data method to load data from files, Redshift, PostgreSQL, or SQLite databases.

4. Ask a question
   Use the ask method to pose a question and generate visualizations, insights, recommendations, and tasks.

Let's review these steps individually for Pythonic data analysis.

Installation

The first step is to install via pip. To access aih's data analysis features, install it with the data-analysis variant.

pip install aih[data-analysis]
 

This will install the aih package alongside all dependencies required for data analysis.

Creating an Instance

The next step is to create a class instance.

from aih import DataAnalysis
analysis = DataAnalysis(analysis_type="ml", api_key="openai_key")

The analysis_type parameter accepts three options:

• ml – for analysis using Python code with libraries such as Pandas and Scikit-learn.
• sql – for SQL-based analysis.
• skip – to bypass analysis and obtain insights directly from the uploaded data.

Loading Data

DataAnalysis offers various ways to connect with your data. Whether you're working with data files in formats like CSV, Excel, or JSON, connecting to an online Redshift database, or using a local SQLite database, DataAnalysis has you covered. The get_data class method facilitates data connection and requires three parameters: db_type, db_config, and vector_store_config. The values for these parameters depend on the format of your input data.

Let’s look at a few examples:

Loading Data from Files

Organize all your data files into a list of dictionaries, including their names, paths, and keyword arguments. Then, pass this list to the get_data method when calling it.

db_config = {
    "datasets": [
        {
            "name": "Name of dataset",
            "value": "path/to/dataset.csv",
            # files can be in .csv, .xlsx, .xls, and .json formats
        },
        {
            "name": "Name of dataset",
            "value": "path/to/dataset.xlsx",
            "kwargs": {"sheet_name": "Sheet1"},
            # pass additional keyword arguments for reading the file
        },
        {
            "name": "Name of dataset",
            "value": pd.read_csv("path/to/dataset.csv"),
            # you can also pass pandas DataFrame objects
        },
    ]
}
# Call the get_data method
analysis.get_data(
    db_type = "files",
    db_config = db_config,
)

The value of db_type tells the system:

• Which type of data it will need to explore.
• What to expect in the db_config parameter.

Loading Data from Redshift

First, you will need to collect all the Redshift details.

redshift_config = {
    "host": "",
    "port": 3000, # e.g. 5439
    "database": "",
    "user": "",
    "password": "",
    "schema": ["schema1", "public"], # optional
    "tables": ["table1", "table2"], # optional
}
analysis.get_data(
    db_type = "redshift",
    db_config = redshift_config,
    vector_store_config = { # optional
        "path": "path/to/vector/store",
        "remake": True, # overwrite the existing vector store, if any
    }
)

Once again, note that the value of db_type tells the system:

• Which type of data it will need to explore.
• What to expect in the db_config parameter.

Although you can specify only one database in db_config, there is no limit to the number of tables and schemas, which should be provided as lists. If no schemas and tables are specified, all tables from the public schema will be included by default.

Loading Data from PostgreSQL

The implementation for PostgreSQL closely mirrors that of Redshift. Begin by gathering all the database details.

postgres_config = {
    "host": "",
    "port": 3001, # e.g. 5439
    "database": "",
    "user": "",
    "password": "",
    "schema": ["schema1", "public"], # optional
    "tables": ["table1", "table2"], # optional
}
analysis.get_data(
    db_type = "postgres",
    db_config = postgres_config,
)

Loading Data from SQLite

You can also use a local SQLite database for analysis with DataAnalysis. Simply provide the path to the database in the db_config parameter.

analysis.get_data(
    db_type = "sqlite",
    db_config = {"db_path": "path/to/sqlite.db"},
)

Alternatively, if you have a URL which holds the SQLite database, you can pass this URL as the value of db_path.

Getting Results

You can utilize the DataAnalysis object to analyze a DataFrame by submitting an analysis query to the ask method. This function lets you pose questions directly related to your data, enabling DataAnalysis to handle the query and deliver the appropriate visualizations, insights, recommendations, and tasks.

The simplest implementation looks like this:

result = analysis.ask("Your question here") # returns dict

Here, result has keys visualisation, insights, recommendations, and tasks.

You can control the outputs received from ask:

result = analysis.ask(
    user_input = "Your question here",
    outputs = ["insights"],
)

Here, result still has the keys visualisation, insights, recommendations, and tasks but their values are changed.

>>> print(result["insights"])
This is the insights generated by the LLM analysis...
>>> print(result["recommendations"])
 
>>> repr(result["recommendations"])
"''"
 
You can also define the context for the analysis and output generation. For instance, you might indicate that the analysis is intended for a particular user profile or specify how the outputs should be presented.
 
#### An Example
 
```python
result = analysis.ask(
    "How many columns in my dataset?",
    context = {"insights": "Give your response as a statesman."},
    outputs = ["insights"],
)
print(result['insights'])

• Your dataset contains a total of 11 columns, esteemed colleague..
• The dataset comprises 147 rows, with ranks ranging from 1 to 147.
• The mean rank stands at 74, representing the midpoint of the dataset.

Obtaining Visualizations

To obtain plots from your analysis, use the ask method and specify visualisation in the outputs parameter. The returned dictionary will include a key called visualisation that provides the path to the PNG image. By default, visualization images are saved to ./generated_plots/plot.png, but you can adjust this location using the plot_path parameter.

Here’s an example:

result = analysis.ask(
    "Your query here",
    outputs = ["visualisation"],
    plot_path = "my_image.png",
)

The result then has a visualisation key which is simply the value of plot_path:

>>>> print(result["visualisation"])
my_image.png

It is also possible to pass a directory to plot_path. The generated image is then saved in the directory. Additionally, you may pass a context for the image generation.

result = analysis.ask(
    "Your query here",
    outputs = ["visualisation"],
    context = {"visualisation": "Your visualisation context here"},
    plot_path = "my_directory",
)
print(result["visualisation"])
> my_directory/plot.png

You can then view the image using Pillow or Matplotlib:

from PIL import Image
image = Image.open(result["visualisation"])
image.show()

or

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
img = mpimg.imread(result["visualisation"])
imgplot = plt.imshow(img)
plt.axis('off')
plt.show()