Choose schema sources
Introduction
When you create transformations, you need to decide how to define your source and target schemas. This page explains your options.
You can define schemas in two ways:
- Activity-defined schemas: Schemas that connector activities generate automatically.
- Transformation-defined schemas: Custom schemas that you create within the transformation.
Understanding when to use each approach helps you build transformations that are easier to maintain and more flexible. For hands-on practice with these concepts, see the Create your first transformation.
Activity-defined schemas
Activity-defined schemas are created automatically by connector activities. These activities include database queries, API calls, or file operations. These schemas inherit their structure from the connected system.
Activity-defined schemas are beneficial for these scenarios:
- Your connectors support automatic generation. Examples include database activities, Salesforce, NetSuite, and ServiceNow.
- You want automatic updates and reduced maintenance. Source system changes update transformations automatically and keep endpoints synchronized.
- You're working with standard, established data formats. This includes database schemas, API specifications, and enterprise applications.
Activity-defined schemas follow this process:
- Configure your activity: Set up database connections, API endpoints, or file locations.
- Schema generation: The activity automatically generates request schemas, response schemas, or both.
- Schema inheritance: Adjacent transformations inherit these schemas automatically.
- Precedence: Activity schemas take priority over transformation-defined schemas.
Example
A Database Query activity automatically generates a response schema based on your SQL query results. When you add a transformation after this activity, it inherits this schema as its source.
Activity schema types
Activities generate schemas in two ways:
-
Connector-generated schemas: Generated directly from endpoint metadata and update automatically when you refresh the activity. Examples include database table structures and API response formats.
-
User-defined schemas in activities: Custom schemas that you upload during activity configuration and remain fixed unless you manually update them. Examples include sample JSON files for HTTP activities or CSV templates for file activities.
Transformation-defined schemas
Transformation-defined schemas are custom schemas that you create within the transformation itself. You have full control over their structure and content.
Transformation-defined schemas are beneficial for these scenarios:
- You need specialized structures not provided by standard connectors. Examples include custom output formats, combining multiple data sources, and specialized applications.
- Your endpoints don't provide automatic schemas. This includes legacy systems, custom APIs without specifications, and manual file formats.
- You want schemas that won't change with external updates. Use this approach for standardized formats, shared integrations, and when you need direct control over structure.
You can create transformation schemas using these methods:
| Schema type | Description | Best for |
|---|---|---|
| Custom flat | Single-level structures like CSV files. | Tabular data and straightforward mapping. |
| Custom hierarchical | Complex, nested structures like XML or JSON. | Multiple levels, relationships, and complex data organizations. |
| Sample file | Generate schemas from existing data files (CSV, JSON, XML, XSD). | Automatically detecting structure and data types from examples. |
| Mirrored | Copy structure from the other side of your transformation. | When source and target have similar formats that need customization. |
Note
Consider these precedence rules when you are building your operations:
- Adjacent activity schemas override transformation schemas.
- Even if you move an activity, its schema remains inherited.
- Transformation schemas are used by default only when no activity schema is available.
Example scenario
In this operation, schema precedence works as follows:
- Source schema: Inherited from the Database Query activity (activity-defined).
- Target schema: Inherited from the HTTP v2 POST activity (if schema provided) or transformation-defined. If you remove the HTTP v2 POST activity, the transformation keeps the target schema as a transformation-defined schema.
Update schemas from endpoints
When external systems change, you must regenerate schemas to use those changes in your project. This may be necessary when you:
- Add custom fields to a system.
- Upgrade to a new API version.
- Modify database table structures.
Update connector-generated schemas
Connector-generated schemas get their structure from the endpoint's API. When the endpoint changes, you must refresh the schema in Integration Studio.
For all connectors except Database, NetSuite, Salesforce, Salesforce Service Cloud, and ServiceMax you can regenerate the schema from two locations:
-
Activity configuration: Open the activity and navigate to the Data Schema step and click Refresh.
-
Transformation: Hover on the source or target structure and click Refresh schema.
The schema updates from the endpoint and refreshes in all locations where it's referenced.
Database, NetSuite, and certain Salesforce connectors
The Database, NetSuite, Salesforce, Salesforce Service Cloud, and ServiceMax connectors require reconfiguration to refresh schemas:
- Open the activity configuration.
- Advance through each configuration step.
- Make at least one change, such as adding and removing a character from the activity name.
- Complete the configuration.
The schema refreshes when you complete the configuration steps.
Note
When you refresh a schema, all instances of that schema throughout your project are updated. This includes all transformations and activities that reference the schema.
Update transformation-defined schemas
Changes to user-defined schemas automatically propagate to all references. When you edit a schema used in multiple places, a dialog shows affected components:
Dialog text
Schema used by multiple components
This schema is used by multiple components. Any changes made to it might affect the validity of these components: - Component 1 - Component 2
To prevent changes from affecting other components, first remove or replace the schema reference in those components.
Handle schema changes in operations
If an endpoint's schema changes but you haven't regenerated it in Integration Studio, your operations may fail. The impact depends on:
- Which fields changed.
- Whether changed fields are mapped.
- Whether changed fields are required.
For example:
- Removed required field: Operations fail with errors from the endpoint.
- Removed optional field: Operations may succeed if the field isn't mapped.
- Added required field: Operations fail if the field isn't mapped.
- Added optional field: Operations succeed but new field is ignored.
Tip
Regenerate schemas whenever you modify endpoint systems to prevent runtime errors.
Legacy project migration
Projects created before Harmony version 10.33.2 automatically rename user-defined schemas using this pattern:
<Original Schema Name>_<Transformation Name>_<Source or Target>
For example, CustomerData becomes CustomerData_TransformCustomers_Source.
This naming ensures unique schema identification across transformations.