Unstructured Datastore - Pinkfish - AI Agents & Workflows for Getting Work Done

Overview

Unstructured Datastores provide flexible NoSQL storage for any data type without schema requirements. Store strings, numbers, objects, or arrays - perfect for dynamic content, logs, and API responses.

Creating a Datastore

API Endpoint

POST /datacollection/

Request Body

{
  "name": "My Datastore",
  "description": "Optional description",
  "type": "datastore",
  "createdBy": "your-provider-id",
  "createdByName": "Your Name",
  "triggerUrls": []
}

Parameters:

name (required) - Display name for your datastore
description (optional) - Description of what this datastore contains
type (required) - Must be “datastore” for unstructured datastores
createdBy (required) - Your provider ID
createdByName (required) - Your display name
triggerUrls (optional) - Collection-level event triggers
isStructured (optional) - Must be false or omitted for unstructured datastores

Response:

{
  "id": "abc123-collection-id",
  "message": "Data collection created successfully"
}

Note: By default, collections are created as unstructured unless isStructured: true is explicitly set.

Key Features

No schema required - store any JSON structure
Flexible content - strings, numbers, objects, arrays
Fast search - fuzzy, exact, and prefix matching
Event triggers - automate workflows on data changes
Batch operations - create up to 500 items at once
Multiple access methods - by key, key+sortField, or itemId

Basic Command Structure

/selected-datastore [action] [parameters]

CRUD Operations

Create Item

/selected-datastore create item with:
key: user-settings
sortField: theme-preferences
content: {"theme": "dark", "language": "en", "notifications": true}
metadata: {"version": "1.0", "lastSync": "2024-01-15"}

Parameters:

key (required) - Primary identifier
content (required) - Any JSON structure
sortField (optional) - Auto-generated if not provided
metadata (optional) - Custom tracking attributes
triggerUrls (optional) - Event-driven webhooks

Batch Create

Create up to 500 items efficiently:

/selected-datastore create batch with items:
[
  {
    "key": "logs",
    "sortField": "2024-01-15-10:30",
    "content": {"level": "info", "message": "Server started"},
    "metadata": {"service": "api"}
  },
  {
    "key": "logs",
    "sortField": "2024-01-15-10:31",
    "content": {"level": "warn", "message": "High memory usage"},
    "metadata": {"service": "api"}
  }
]

Best practices:

Keep batches under 100 items for best performance
Larger batches (up to 500) may cause gateway timeouts
Each item can have different structure

Read Items

By item ID (recommended):

/selected-datastore get item with itemId: abc123def456

By key (returns all items with that key):

/selected-datastore get items with key: user-settings

By key + sortField (returns specific item):

/selected-datastore get item with:
key: user-settings
sortField: theme-preferences

List Items

/selected-datastore list items with:
limit: 50
orderedBy: createdAt:desc
format: light

Parameters:

limit - Max results (max: 1000, returns all if not specified)
orderedBy - Sort by createdAt or updatedAt (e.g., “createdAt:desc”)
format - Set to “light” for minimal data (excludes content)

Update Item

Full Replace (PUT) - Recommended for unstructured datastores Replaces the entire content field:

PUT /memory/items/{itemId}?collectionId=xxx

{
  "content": "new string value"
}

Or with object content:

{
  "content": {"theme": "light", "notifications": false},
  "metadata": {"version": "1.1"}
}

Content is completely replaced. Works with any data type (string, number, object, array). Partial Update (PATCH) Only for object content - merges with existing:

PATCH /memory/items/{itemId}?collectionId=xxx

{
  "content": {"theme": "light"}
}

Existing fields in content object are preserved. Only works with object content (not strings, numbers, or arrays).

Delete Item

By itemId (recommended):

/selected-datastore delete item with itemId: abc123def456

By key + sortField:

/selected-datastore delete item with:
key: user-settings
sortField: theme-preferences

By key (deletes all items with that key):

/selected-datastore delete items with key: user-settings

Search

Advanced search with fuzzy matching:

/selected-datastore search with:
q: customer feedback
field: content
type: fuzzy
threshold: 0.3
limit: 50

Parameters:

q (required) - Search query
field - Where to search: “all” (default), “content”, “key”, “sortField”. Note: “all” also searches metadata content.
type - Match type: “fuzzy” (default), “exact”, “prefix”
threshold - Fuzzy sensitivity (0.0 strict to 1.0 lenient, default 0.3)
limit - Max results (default: 50, max: 1000)
compiled - Output as plain text (true) or JSON with snippets (false)

Response with snippets:

{
  "results": [
    {
      "docId": "abc123",
      "fieldMatched": "content",
      "matchedSnippet": "...customer **feedback** on the new...",
      "score": 0.84,
      "relevance": "high"
    }
  ]
}

Filtering by Metadata

You can filter search results (and list queries) by metadata fields using dot notation in query parameters. This is useful for narrowing results by tags, categories, status, or any custom tracking attributes you’ve stored in metadata. Filter by metadata field:

/selected-datastore search with:
q: customer feedback
metadata.status: active
metadata.category: support

API equivalent:

GET /memory/items?collectionId=xxx&metadata.status=active&metadata.category=support

How it works:

Metadata filters use exact matching (not fuzzy)
Multiple metadata filters are combined with AND logic
Filters are applied before text search, reducing the search space
Works with both search and list operations

Example use cases: Filter logs by service:

/selected-datastore search with:
q: connection error
metadata.service: api-gateway
metadata.environment: production

Filter by custom tags:

/selected-datastore list items with:
metadata.priority: high
metadata.assignee: alice
limit: 100

Best practice: Use metadata for filterable attributes that you’ll query by, such as:

status - active, archived, pending
category - support, sales, engineering
priority - high, medium, low
tags - comma-separated or individual fields
environment - production, staging, development
version - 1.0, 2.0

This keeps your keys clean for primary access patterns while enabling flexible filtering through metadata.

Key and Sort Field Patterns

The key and sortField combination in Pinkfish datastores works similarly to partition keys (PK) and sort keys (SK) in AWS DynamoDB. Understanding these patterns unlocks powerful data organization and query capabilities.

How It Works

Concept	Pinkfish	DynamoDB Equivalent
Primary grouping	`key`	Partition Key (PK)
Secondary sorting/uniqueness	`sortField`	Sort Key (SK)
Unique identifier	`key` + `sortField`	Primary Key

Key rules:

The key + sortField combination must be unique within a collection
Items with the same key are grouped together and can be queried as a set
sortField determines ordering within a key group
If sortField is omitted, one is auto-generated (timestamp-based)

Access Patterns

You can retrieve items in three ways:

By itemId (direct lookup, fastest)

/selected-datastore get item with itemId: abc123def456

By key (returns ALL items with that key)

/selected-datastore get items with key: USER#12345

By key + sortField (returns specific item)

/selected-datastore get item with:
key: USER#12345
sortField: ORDER#2024-01-15

Pattern 1: Hierarchical Entity Keys

Use compound keys with delimiters (# is common) to represent entity relationships:

key: ORG#acme-corp
sortField: USER#[email protected]

key: ORG#acme-corp
sortField: USER#[email protected]

key: ORG#acme-corp
sortField: DEPT#engineering

Query all items for an organization:

/selected-datastore get items with key: ORG#acme-corp

This returns all users, departments, and other entities under that organization. Use cases: Multi-tenant applications, organizational hierarchies, project structures

Pattern 2: One-to-Many Relationships

Store parent and child entities together by sharing a key:

key: CUSTOMER#cust-789
sortField: PROFILE
content: {"name": "Alice Johnson", "email": "[email protected]"}

key: CUSTOMER#cust-789
sortField: ORDER#2024-001
content: {"total": 150.00, "status": "shipped"}

key: CUSTOMER#cust-789
sortField: ORDER#2024-002
content: {"total": 89.99, "status": "pending"}

key: CUSTOMER#cust-789
sortField: ADDRESS#shipping
content: {"street": "123 Main St", "city": "Seattle"}

Query all customer data:

/selected-datastore get items with key: CUSTOMER#cust-789

Returns profile, all orders, addresses - everything related to that customer. Use cases: Customer records with orders, users with preferences, products with reviews

Pattern 3: Time-Series Data

Use timestamps or date prefixes in sortField for chronological ordering:

key: LOGS#api-server
sortField: 2024-01-15T10:30:00Z
content: {"level": "info", "message": "Server started"}

key: LOGS#api-server
sortField: 2024-01-15T10:31:15Z
content: {"level": "warn", "message": "High memory usage"}

key: LOGS#api-server
sortField: 2024-01-15T10:32:00Z
content: {"level": "error", "message": "Connection timeout"}

Query all logs for a service:

/selected-datastore get items with key: LOGS#api-server

Items return sorted by timestamp (sortField ordering). Use cases: Event logs, audit trails, activity feeds, metrics history

Pattern 4: Type Prefixes for Mixed Entities

Prefix sortField with entity type to organize different record types under one key:

key: PROJECT#proj-123
sortField: META#info
content: {"name": "Website Redesign", "status": "active"}

key: PROJECT#proj-123
sortField: TASK#task-001
content: {"title": "Create wireframes", "assignee": "alice"}

key: PROJECT#proj-123
sortField: TASK#task-002
content: {"title": "Design mockups", "assignee": "bob"}

key: PROJECT#proj-123
sortField: COMMENT#2024-01-15T10:00:00Z
content: {"author": "alice", "text": "Looking good!"}

Query entire project:

/selected-datastore get items with key: PROJECT#proj-123

Use cases: Project management, document systems, content with metadata

Pattern 5: Composite Sort Keys

Combine multiple attributes in sortField for rich querying:

key: INVENTORY
sortField: CATEGORY#electronics#SKU#12345
content: {"name": "Wireless Mouse", "price": 29.99, "stock": 150}

key: INVENTORY
sortField: CATEGORY#electronics#SKU#12346
content: {"name": "USB Keyboard", "price": 49.99, "stock": 75}

key: INVENTORY
sortField: CATEGORY#clothing#SKU#20001
content: {"name": "T-Shirt", "price": 19.99, "stock": 200}

This enables future filtering by category while keeping all inventory queryable together. Use cases: Product catalogs, file systems, categorized content

Pattern 6: User-Scoped Data

Isolate data per user while maintaining consistent structure:

key: USER#user-456#SETTINGS
sortField: preferences
content: {"theme": "dark", "language": "en", "notifications": true}

key: USER#user-456#BOOKMARKS
sortField: 2024-01-15T09:00:00Z
content: {"url": "https://example.com", "title": "Example Site"}

key: USER#user-456#BOOKMARKS
sortField: 2024-01-16T14:30:00Z
content: {"url": "https://docs.pinkfish.ai", "title": "Pinkfish Docs"}

Query user settings:

/selected-datastore get items with key: USER#user-456#SETTINGS

Query user bookmarks:

/selected-datastore get items with key: USER#user-456#BOOKMARKS

Use cases: User preferences, personal data, per-user collections

Design Best Practices

Choosing your key:

Group items you’ll query together under the same key
Include entity type prefixes for clarity (USER#, ORDER#, ORG#)
Consider your most common access pattern

Choosing your sortField:

Use for uniqueness within a key group
Include timestamps for chronological ordering
Use type prefixes to organize mixed entity types
Make it meaningful - you’ll see it when debugging

Common delimiters:

# - Most common (e.g., USER#123#ORDER#456)
_ - Alternative (e.g., user_123_order_456)
/ - Path-like (e.g., users/123/orders/456)

Avoid:

Overly long keys (keep under 256 characters)
Unpredictable or random-only keys when you need to query by group
Storing unrelated items under the same key

Key Immutability

Keys and sort fields cannot be updated after creation. This is a fundamental characteristic of NoSQL databases like DynamoDB and applies to Pinkfish datastores as well. Why keys are immutable:

Referential integrity - Other systems may reference items by their key + sortField. Changing keys would break those references silently.
Query consistency - Items are indexed and organized by their keys. Allowing key changes would require re-indexing and could cause items to “disappear” from expected query results.
Atomicity - A key change is really a delete + create operation. Making this explicit prevents accidental data loss and ensures you consciously handle the transition.
Performance - Immutable keys enable efficient indexing and caching strategies that would be impossible with mutable keys.

What this means for your design:

✅ Use immutable values in keys:
   - IDs: USER#user-789, ORDER#ord-12345
   - Timestamps: 2024-01-15T10:30:00Z
   - Email addresses (if stable): [email protected]

❌ Don't put mutable data in keys:
   - Status: STATUS#pending (status changes!)
   - Priority: PRIORITY#high (priority changes!)
   - Counts: COUNT#5 (counts change!)

Put mutable data in content or metadata: Both content and metadata fields are fully mutable and can be updated at any time.

key: TASK#task-001
sortField: DETAILS
content: {
  "title": "Design homepage",
  "description": "Create the main landing page"
}
metadata: {
  "status": "in-progress",    ← can be updated, filterable
  "priority": "high",         ← can be updated, filterable
  "assignee": "alice"         ← can be updated, filterable
}

When to use metadata vs content:

metadata - Filterable attributes you’ll query by (status, priority, tags, category)
content - The main data payload (documents, configurations, records)

Metadata fields can be filtered using metadata.fieldName=value in queries, making them ideal for attributes you need to search or filter by. If you need to change a key or sortField: The standard pattern is delete + create:

Read the existing item (by itemId or key+sortField)
Create a new item with the desired key/sortField
Delete the original item

This makes the operation explicit and allows you to handle any side effects (updating references, triggering notifications, etc.). The itemId is always stable: Every item has an auto-generated itemId that never changes. Use this when you need a permanent reference to an item regardless of its key or sortField:

{
  "itemId": "abc123def456",  ← permanent, never changes
  "key": "USER#user-789",
  "sortField": "PROFILE",
  "content": {...}
}

Event Triggers

Automatically start workflows when data changes in your unstructured datastore. Triggers can be set at both collection-level (all items) and item-level (specific items). Supported events:

onCreate - New items added
onEdit - Existing items modified
onDelete - Items removed

Trigger levels:

Collection-level - Set via the collection management UI, fires for all items in the collection
Item-level - Set via triggerUrls parameter when creating/updating items, fires only for that specific item

For complete documentation on setting up triggers, payload formats, rate limiting, and best practices, see the Event Triggers guide.

Common Use Cases

Application logs and event tracking
API response caching
Configuration and settings storage
Dynamic content management
Queue systems and task processing
Session data storage
Temporary data and cache
Flexible data structures that change over time

Best Practices

Key & Sort Field:

Use descriptive keys that group related items
Use sortField for uniqueness within a group
itemId is most efficient for direct access

Content Structure:

Keep content structure consistent within a collection for easier querying
Use metadata for cross-cutting concerns (tags, timestamps, status)

Triggers:

Use collection-level triggers for system-wide monitoring
Use item-level triggers (via triggerUrls) for specific items that need special handling
Set triggerChanges: false on API calls to prevent recursive triggers

Performance:

Use batch operations for multiple creates
Use format: “light” when you only need metadata
Set appropriate limits to reduce payload size

Notes

No schema validation - you can store any JSON structure
Items uniquely identified by key + sortField combination
Auto-generated sortField if not specified
Search supports fuzzy matching for flexible queries
PUT replaces entire content (use for strings, numbers, arrays, or full object replacement)
PATCH merges content (only for object content, preserves unspecified fields)
Collections can hold different content structures per item

Get Started

Organization

Agents

Workflows

Resources

Integrations

Orchestration

Credits & Pricing

Skills

How To Guides

Release Notes

Support

​Overview

​Creating a Datastore

​API Endpoint

​Request Body

​Key Features

​Basic Command Structure

​CRUD Operations

​Create Item

​Batch Create

​Read Items

​List Items

​Update Item

​Delete Item

​Search

​Filtering by Metadata

​Key and Sort Field Patterns

​How It Works

​Access Patterns

​Pattern 1: Hierarchical Entity Keys

​Pattern 2: One-to-Many Relationships

​Pattern 3: Time-Series Data

​Pattern 4: Type Prefixes for Mixed Entities

​Pattern 5: Composite Sort Keys

​Pattern 6: User-Scoped Data

​Design Best Practices

​Key Immutability

​Event Triggers

​Common Use Cases

​Best Practices

​Notes

Overview

Creating a Datastore

API Endpoint

Request Body

Key Features

Basic Command Structure

CRUD Operations

Create Item

Batch Create

Read Items

List Items

Update Item

Delete Item

Search

Filtering by Metadata

Key and Sort Field Patterns

How It Works

Access Patterns

Pattern 1: Hierarchical Entity Keys

Pattern 2: One-to-Many Relationships

Pattern 3: Time-Series Data

Pattern 4: Type Prefixes for Mixed Entities

Pattern 5: Composite Sort Keys

Pattern 6: User-Scoped Data

Design Best Practices

Key Immutability

Event Triggers

Common Use Cases

Best Practices

Notes