Model Composite

Introduction

Model composition is the ability to treat multiple models together as a single entity. This oeCloud feature allows developers to expose certain operations on multiple models as a single API call. It also allows combining non-related models and perform similar operation as done with single model.

There are two types of model composition.

  • Implicit Composite
  • Explicit Composite

This module overrides create, replaceById, updateAttributes and a few other DataAccessObject functions of loopback-datasource-juggler. Each of these function checks for the related model data in the payload. If related model data is present, it executes associated instructions first along with actual model payload.

Implicit Composite

Loopback provides a way to relate one or more models using relations. oeCloud can use these relations to combine multiple operations as a single logical unit of work or a single API call. For example, you can fetch, create or update parent model and it’s children in single Web API call.

Explicit Composite

If models are not related and you wish to get or post data of those unrelated model using single operation, you can construct an new explicit composite model and specify other models that constitute the ingredients of the composite.

Development

$ git clone http://evgit/oecloud.io/oe-model-composite.git
$ cd oe-model-composite
$ npm install --no-optional
$ npm run grunt-cover

You should see coverage report in coverage folder.

Getting Started

In this section, we will see how we can add this module in your project and use this feature.

Installation

$ npm install oe-model-composite --save

Enabling the feature

To enable this feature, you must add an entry into application’s app-list.json file as shown below.

  {
    "path": "oe-model-composite",
    "enabled": true
  }

This automatically enables the composite post on all models/APIs in the application.

Reference Models

Following discussion uses several models to demonstrate the model composite functionality. These models are listed below:

CustomAddress

{
    "name": "CustomerAddress",
    "base": "BaseEntity",
    "idInjection": true,
    "properties": {
        "city": {
            "type": "string",
            "required": true
        },
        "country": {
            "type": "string",
            "required": true
        }
    } 
}

CustomerEmail

{
    "name": "CustomerEmail",
    "base": "BaseEntity",
    "idInjection": true,
    "properties": {
        "email": {
            "type": "string",
            "required": true
        },
        "active": {
            "type": "boolean",
            "required": true
        }
    } 
}

Customer

A Customer is related to CustomerAddress and CustomerEmail with hasMany relation. i.e. A customer has many addresses and has many email IDs.

{
    "name": "Customer",
    "base": "BaseEntity",
    "idInjection": true,
    "properties": {
        "name": {
            "type": "string",
            "required": true
        }
    },
    "relations": {
        "addresses": {
            "type": "hasMany",
            "model": "CustomerAddress",
            "foreignKey": "customerId"
        },
        "emails": {
            "type": "hasMany",
            "model": "CustomerEmail",
            "foreignKey": "customerId"
        }
    }      
}

Promotion

{
    "name": "Promotion",
    "base": "BaseEntity",
    "idInjection": true,
    "properties": {
        "name": {
            "type": "string",
            "required": true
        },
        "active": {
            "type": "boolean",
            "required": true
        }
    } 
}

UpcomingEvent

{
    "name": "UpcomingEvent",
    "base": "BaseEntity",
    "idInjection": true,
    "properties": {
        "eventName": {
            "type": "string",
            "required": true
        },
        "active": {
            "type": "boolean",
            "required": true
        }
    } 
}

Reference Data

  • Create a Customer by posting following data to /api/Customers endpoint.

    {
      "name": "Roald Dahl",
      "id": "C001"
    }
    
  • Add an address to this customer by posting following data to /api/CustomerAddresses endpoint.

    {
      "city": "London",
      "country": "UK",
      "customerId": "C001"
    }
    
  • Add an email Id to this customer by posting following data to /api/CustomerEmails endpoint.

    {
      "email": "roald.dahl@stories.com",
      "active": true,
      "customerId": "C001"
    }
    
  • Upcoming Event Data, define by posting on /api/UpcomingEvents. Note that we are posting two records.

[
    {
        "eventName" : "Property Exhibition",
        "active" :true
    },
    {
        "eventName" : "Webinar on house buying",
        "active" :false
    }
]
  • Promotion Data, POST /api/Promotions
[
    {
        "name" : "Special Interest Rates",
        "active" : true
    }
]

Using Implicit Composite

In the models created above, the Customer model has

  • hasMany relationship with CustomerAddress model (relation name addresses)
  • hasMany relationship with CustomerEMail model (relation name emails)

Fetching Composite Data

You can use include filter to fetch related model data as described in Loopback documentation.

/api/Customers?filter={"include":["addresses","emails"]} fetches all the customers along with their associated addresses and emails.

This is a loopback default functionality and is unrelated to oe-model-composite module.

For our example data above, an API call to api/Customers/C001?filter={"include":["addresses","emails"]} fetches us Customer, CustomerAddress and CustomerEmail records together with a single web API call. Loopback will internally get related addresses and emails for given Customer record and embed it as collection (javascript Array) in response.

{
  "name": "Roald Dahl",
  "id": "C001",
  "addresses": [
    {
      "city": "London",
      "country": "UK",
      "id": "6267c3317e8e6346104c92f0",
      "customerId": "C001"
    }
  ],
  "emails": [
    {
      "email": "roald.dahl@stories.com",
      "active": true,
      "id": "6267c34f7e8e6346104c92f1",
      "customerId": "C001"
    }
  ]
}

Thus, even when there is single GET call to Customer, you get data from three models ( Customer, CustomerAddress and CustomerEmail ).

This is important. Many times, the UI screen is driven by parent model entity. (eg Customer with Customer-Id=1234).

Usually, you would want to bring and show one Customer data and all it’s related child data (eg all addresses, family members, phone numbers, emails etc) on screen.

Creating Composite Data

Typically, the user interaction for capturing Customer and CustomerAddress is to collect the data in UI/browser and make an API call sending all the collected data.

However, when posting entire data on /api/Customers the default loopback behavior is to only save Customer data and ignore any related model data.

Suppose we want to create a Customer with single CustomerAddress and CustomerEmail. With plain Loopback, this will need three API calls as we did manually in Reference Data section.

  • POST /api/Customers
  • POST /api/CustomerAddresses
  • POST /api/CustomerEmails

If there are more child models/records, each will need separate PUT or POST API call.

With oeCloud’s Implicit Composite feature, we can POST entire data along with the related model data and make a single API call, POST /api/Customers

{
    "id": "C002",
    "name": "Enid Blyton",
    "addresses": [{
        "city": "Fremont",
        "country": "USA"
    }],
    "emails": [{
        "active": true,
        "email": "enid@storyteller.in"
    }]
}

As shown in above example, if you post the data as above, Customer, CustomerAddress and CustomerEmail - all of these models will be populated with data posted. Relations will be taken into account and in database, you could see foreign keys being populated in child tables. The API response would be similar to below,

{
  "name": "Enid Blyton",
  "id": "C002",
  "addresses": [
    {
      "city": "Fremont",
      "country": "USA",
      "id": "6267ec7f7e8e6346104c92f2",
      "customerId": "C002"
    }
  ],
  "emails": [
    {
      "email": "enid@storyteller.in",
      "active": true,
      "id": "6267ec7f7e8e6346104c92f3",
      "customerId": "C002"
    }
  ]
}

Modifying Composite Data

To update the parent and child records together, you must use the PUT operation with appropriate indicators about how to handle the individual child record.

Using implicit composite, the parent model operation is implicitly defined as modify. However for child models, you must set operation explicitly using __row_status field.

Consider sample PUT operation payload below. We are achieving following modifications to our Customer.

  • Modify the customer name
  • Delete existing CustomerAddress and add a new one
  • Modify existing CustomerEmail and add a new one
{
  "name": "Enid Blyton (The Author)",
  "id": "C002",
  "addresses": [
    {
      "city": "Fremont",
      "country": "USA",
      "id": "6267ec7f7e8e6346104c92f2",
      "customerId": "C002",
      "__row_status": "deleted"
    },
    {
      "city": "Sidney",
      "country": "Australia",
      "__row_status": "added"
    }
  ],
  "emails": [
    {
      "email": "enid@storyteller.in",
      "active": false,
      "id": "6267ec7f7e8e6346104c92f3",
      "customerId": "C002",
      "__row_status": "modified"
    },
    {
      "email": "enid@stories.au",
      "active": true,
      "__row_status": "added"
    }
  ]
}

This will have a response data similar to below. As intended, the Customer name is modified, existing CustomerAddress is deleted and a new one is added. Also, the existing CustomerEmail is marked as active: false (modification) and a new email record is added.

{
  "name": "Enid Blyton (The Author)",
  "id": "C002",
  "addresses": [
    {
      "city": "Sidney",
      "country": "Australia",
      "id": "6267ef247e8e6346104c92f4",
      "customerId": "C002"
    }
  ],
  "emails": [
    {
      "email": "enid@storyteller.in",
      "active": false,
      "id": "6267ec7f7e8e6346104c92f3",
      "customerId": "C002"
    },
    {
      "email": "enid@stories.au",
      "active": true,
      "id": "6267ef247e8e6346104c92f5",
      "customerId": "C002"
    }
  ]
}

Using Explicit Composite

Consider landing page of a typical banking website. When you see your home screen, usually you see following

  • Your profile name and other details (coming from UserProfile model)
  • All your accounts with account types (savings, current, FD, loan accounts etc) and account balance ( coming from account + accountBalance models)
  • Notification and reminders (coming from notification model)
  • Your last n transactions (coming from transaction Model )
  • List of bank offers and promotions (coming from promotion model)
  • List of upcoming events like webinars (coming from upcomingEvents model)

There are several ways this can be achieved.

  • Make multiple API calls to pull different data. This would be costly and prohibitive from performance point of view.
  • Writing a custom remote endpoint, programmatically prepare the composite response and return.
  • Using explicit composite functionality to declaratively make the composite data available for front-end.

Out Customer model has many addresses and emails. There are unrelated models like Promotion and UpcomingEvents. If you want to construct API that returns all the data in a single API call, you need to create a explicit-composite model.

Composite model definition is shown below. It consists of three models Customer, Promotions and UpcomingEvents.

Defining Explicit Composite Model

Create a new model HomePageData as below,

{
  "name": "HomePageData",
  "CompositeTransaction": true,
  "compositeModels": {
    "Customer": {},
    "Promotion": {},
    "UpcomingEvent": {}
  },
  "properties": {},
}

The APIs exposed by HomePageData model would now accept and return Customer, Promotion and UpcomingEvent data under respective properties.

Fetching Data on Composite Model

GET /api/HomePageData can now be used to fetch composite data with appropriate filter. Make GET /api/HomePageData?filter={...} API call with filter object as below, will return Customer details (including addresses and email Ids) along with all the active Promotions and UpcomingEvents

{
  "Customer": {
    "where":{
      "id": "C001"
    }, 
    "include": ["addresses", "emails"]
  },
  "Promotion" : {
    "where" : {
      "active" : true
    }
  },
  "UpcomingEvent" : {
    "where" : {
      "active" : true
    }
  }
}

Note the format of filter object. Filter object would contain property having name of the Model in composite definition. In our case, it would have three keys namely Customer, Promotion and UpcomingEvent. The value for each key is a complete filter definition as supported by loopback. Here, customer object has filter where clause which returns record for customer id C001. Also include clause to include addresses and emails. Promotion and UpcomingEvent has filter to ensure active:true records. This will return data of all the models defined in composite.

For the sample data above, response would look like

{
  "Customer": [
    {
      "name": "Roald Dahl",
      "id": "C001",
      "addresses": [
        {
          "city": "London",
          "country": "UK",
          "id": "6267c3317e8e6346104c92f0",
          "customerId": "C001"
        }
      ],
      "emails": [
        {
          "email": "roald.dahl@stories.com",
          "active": true,
          "id": "6267c34f7e8e6346104c92f1",
          "customerId": "C001"
        }
      ]
    }
  ],
  "Promotion": [
    {
      "name": "Special Interest Rates",
      "active": true,
      "id": "62682e42cd6d5764d43a80ec"
    }
  ],
  "UpcomingEvent": [
    {
      "eventName": "Property Exhibition",
      "active": true,
      "id": "62682db15e902f1f94592578"
    }
  ]
}

Modifying Composite Data

An explicit composite model is simply an assembly of unrelated data. Each of this unrelated data may have its own business process for creation, modification and deletion. Accordingly, in general, these unrelated data would be managed using individual APIs. However, the explicit composite allows managing these unrelated data with a single API call.

The key is, you must tell how each record should be handled by specifying __row_status field for each record. Consider following post data.

{
  "Customer": [
    {
      "name": "The Roald Dahl",
      "id": "C001",
      "__row_status": "modified",
      "addresses": [
        {
          "__row_status": "modified",
          "city": "Oxford",
          "country": "UK",
          "id": "6267c3317e8e6346104c92f0",
          "customerId": "C001"
        }
      ]
    }
  ],
  "Promotion": [
    {
      "__row_status": "deleted",
      "name": "Special Interest Rates",
      "active": true,
      "id": "62682e42cd6d5764d43a80ec"
    },
    {
      "__row_status": "added",
      "name": "Zero Annual Fee",
      "active": true
    }
  ],
  "UpcomingEvent": [
    {
      "__row_status": "modified",
      "eventName": "Property Exhibition (Cancelled)",
      "active": false,
      "id": "62682db15e902f1f94592578"
    }
  ]
}

In payload above,

  • we are modifying the name in main Customer record
  • modifying the city in one of the CustomerAddress
  • deleting a Promotion and adding a new one
  • modifying an UpcomingEvent

Each of these actions are marked in the payload by means of __row_status field having value added, modified or deleted.

This entire operation would run in single transaction and if any of it fails, the transaction would be rolled back.

Transaction and Transaction timeouts

  • This module forces transactions to ensure data consistency. Therefore if a child model data is invalid the whole http implicit POST request (that creates model instances) will be rollback.

  • This module will not lock database tables making it unavailable for other database connections to utilize.

  • However, if other applications (external) do lock tables, we can fail requests using the transactionTimeout application configuration parameter. It needs to be added in the server/config.json file. It expects value in milliseconds. Without this setting transactions won’t fail, and http requests will indefinitely wait until the db table lock is released.