Synadia Connectors: Transformers

You’ve got your inlets and outlets with data flowing smoothly, but now you’re thinking: what if I need to reshape, enrich, or completely transform this data before it reaches its destination?

Welcome to transformers—the powerful middleman that sits between your data sources and destinations, giving you complete control over how your information gets processed.

In this guide, we’ll explore what transformers are, when to use them, and walk through practical examples of both mapping and service transformers that will unlock the full potential of your data streams.

What Are NATS Transformers?

Transformers are processing components that intercept messages within your data pipeline. They allow you to modify, enrich, or restructure data as it flows through your inlet and outlet connectors.

Where Transformers Live

Transformers sit strategically in your data pipeline:

• In outlets: Between your consumer and sink
• In inlets: Between your source and producer

Messages flow into the transformer, transformations are applied, and the modified data continues to the next stage.

Two Types of Transformers

NATS provides two main types of transformers, each suited for different use cases:

1. Mapping Transformers
Built-in transformers using Bloblang to reformat, convert, filter, or reshape data. Fast, lightweight, with no external dependencies. Perfect for inline transformations.

2. Service Transformers
External services that handle complex operations like database lookups, API calls, and custom business logic. Write in your preferred language and deploy wherever needed.

Mapping Transformers: Inline Data Transformation

When to Use Mapping Transformers

Your connector is sending data, but what if it arrives in the wrong format? Mapping transformers reshape, filter, and enrich your data in real-time as it moves through your pipeline.

Key Advantages:

• Everything happens inline—no separate microservices
• No additional processors or extra hops
• Transformation logic lives directly on your connector
• Powered by Bloblang, NATS’s transformation language

Perfect For:

• Format conversions
• Conditional logic
• Field calculations
• Data enrichment
• Message filtering
• Schema validation

Example: Transforming Data for Google Cloud Bigtable

The Problem: You’re sending temperature sensor data to Google Cloud Bigtable, but the data format doesn’t match what Bigtable expects. Your connector logs show “Invalid message format” errors.

The Solution: A mapping transformer that restructures your messages on the fly.

Original Message Format:

1
{
2
  "timestamp": "2024-10-23T10:30:00Z",
3
  "temperature": 72.5,
4
  "sensor_id": "sensor-001"
5
}

Required Bigtable Format:

1
{
2
  "key": "2024-10-23T10:30:00Z",
3
  "data": {
4
    "temperature": 72.5,
5
    "sensor_id": "sensor-001"
6
  }
7
}

The Bloblang Transformation:

1
root.key = this.timestamp
2
root.data = this

This simple code:

1. Takes the timestamp field and uses it as the key
2. Wraps the entire original message in a data object
3. Creates the structure Bigtable expects

Setup Highlights:

1. Create your Google Cloud Bigtable instance and table
2. Add a column family to organize your data
3. Create service account credentials
4. Set up your Bigtable outlet connector
5. Click “Add Transformer” and select “Mapping Transformer”
6. Add your Bloblang code
7. Update connector configuration to use transformed fields

The Result: Data flows seamlessly into Bigtable with the correct format, all transformed inline without any external services.

Service Transformers: Complex Logic Anywhere

When to Use Service Transformers

Mapping transformers are great for inline processing, but sometimes you need more power. Service transformers handle what mapping transformers can’t:

What Service Transformers Enable:

• Accessing data from anywhere (edge devices, databases behind firewalls, restricted APIs)
• Complex logic in any programming language
• Heavy computation that would block inline processing
• Independent scaling from your connectors
• Deployment flexibility (edge, cloud, or on-premises)

The Architecture: Service transformers are built on NATS microservices. They listen on a NATS subject, receive messages, apply transformations, and respond with the modified data. All via NATS messaging.

Example: Cleaning MongoDB Change Stream Data

The Problem: You’re using a MongoDB inlet to capture database changes, but the data includes excess metadata from MongoDB’s change stream format. You want clean, simple messages.

MongoDB Change Stream Output (with metadata):

1
{
2
  "_id": {...},
3
  "operationType": "insert",
4
  "clusterTime": {...},
5
  "fullDocument": {
6
    "timestamp": "2024-10-23T10:30:00Z",
7
    "temperature": 72.5,
8
    "sensor_id": "sensor-001"
9
  },
10
  "ns": {...},
11
  "documentKey": {...}
12
}

Desired Clean Output:

1
{
2
  "timestamp": "2024-10-23T10:30:00Z",
3
  "temperature": 72.5,
4
  "sensor_id": "sensor-001",
5
  "category": "warm"
6
}

The Solution: A Python service transformer that extracts the data you need and adds intelligent categorization.

How It Works:

1. Python microservice connects to NATS
2. Subscribes to temperature.analyze subject
3. Receives raw MongoDB messages
4. Extracts the fullDocument field
5. Classifies temperature as “cold,” “warm,” or “hot”
6. Adds category field to the cleaned message
7. Responds with the transformed data

Sample Python Code Structure:

1
# NATS microservice listening on temperature.analyze
2
def transform_message(msg):
3
    # Parse MongoDB change stream
4
    raw_data = json.loads(msg.data)
5

6
    # Extract the actual document
7
    document = raw_data.get('fullDocument', {})
8

9
    # Classify temperature
10
    temp = document.get('temperature', 0)
11
    if temp < 60:
12
        category = "cold"
13
    elif temp < 80:
14
        category = "warm"
15
    else:
16
        category = "hot"
17

18
    # Add classification
19
    document['category'] = category
20

21
    # Return clean message
22
    return json.dumps(document)

Setup Highlights:

1. Write your transformation service in any language (Python, Go, Java, etc.)
2. Deploy it wherever makes sense (locally, cloud, edge, on-premises)
3. Start your service and ensure it’s connected to NATS
4. In your inlet connector, click “Add Transformer”
5. Select “Service Transformer”
6. Specify the endpoint subject (e.g., temperature.analyze)
7. Restart your connector with the transformer in place

The Result: Clean, enriched data flowing through your inlet with custom business logic applied exactly where you need it.

Key Takeaways

Choose the Right Transformer Type
Use mapping transformers for fast, inline transformations. Use service transformers when you need external data access, complex logic, or language flexibility.

No Overhead, Maximum Flexibility
Mapping transformers add no latency—transformations happen inline. Service transformers leverage NATS messaging for clean, scalable architecture.

Deploy Anywhere
Service transformers can live at the edge, in the cloud, or on-premises. Deploy them wherever your architecture demands.

Language Agnostic
Write service transformers in Python, Go, Java, or whatever fits your team’s expertise and use case.

Production-Ready
Both transformer types integrate seamlessly with connector monitoring, logging, and metrics for full observability.

The Power of Bloblang

Mapping transformers are powered by Bloblang data transformation language. While we’ve shown simple examples here, Bloblang can handle:

• Complex field mapping and restructuring
• Conditional transformations
• Data enrichment
• Filtering and validation
• Type conversions and formatting

Future tutorials will dive deeper into Bloblang’s capabilities for advanced data transformation scenarios.

What’s Next?

Transformers unlock incredible power in your data pipelines, but they’re built on top of an even more fundamental concept: Synadia Workloads. A whole new series is coming to explore workloads in depth, showing you how to build, deploy, and manage microservices in the NATS ecosystem.

Together with inlets and outlets, transformers complete the data processing story—giving you full control over data ingestion, transformation, and delivery in a unified, scalable architecture.