Data ingest from public cloud buckets

hila provides EAI route templates for you to set up routes to ingest files from the following buckets:

Amazon S3
Azure Blob Storage
Google Cloud Storage
Internal data source

First set up your database connection, and then you can set up the route to ingest files from the bucket, and finally ingest into hila from the Data Manager.

Set up your database connection.
1. From User > Settings > Data Source Connections, click Add New Connection to open the Add New Data Source Connection pane.
2. Enter a name, description, URL, and select a connection type.
3. Click Test Connection to test the connection.
4. Click Save.
The connection appears in the Data Source Connections window.
Set up a route to ingest files from the bucket.
1. From User > Settings > Enterprise Application Integration, click the Templates tab.
2. Next to the row with the storage type you want to ingest from, click the route symbol to open the Export to Route pane.
3. Enter a name for the route.
4. Enter values for the fields:
  - Period — how frequently to check the bucket for new files.
  - connectionName — the name of the connection you set up in the previous section.
  - deleteAfterRead — whether to delete the file from the source location after reading it.
  - directory — the directory if the hE folder system to store the files.
5. Click Create New Route.
6. Find the route in the route list and Active toggle to activate the route.
To set up the Data manager to ingest the files from the bucket:
1. In the Data manager window, select an existing folder, or create a new one. This folder is where the files from the bucket are stored.
2. In the right pane, select a bucket connection from the Add from data source pulldown.
3. Enter the name of the data connection in the Bucket field.
4. Enter the directory in the bucket to pull from in the Directory field.
5. Click Add.
The files appear in the folder you specified.

TABLE OF CONTENTS