Tokenization

One of the most important features of the framework is tokenization. Tokenization in the FCF framework is the placement of calculated embedded structures in a text line with the subsequent calculation of the result. Tokenization is performed by the fcf.tokenize() function, which takes two arguments, the first argument is the string to be tokenized, and the second is an object with variables that will be available in the evaluated expressions. Computed constructs are of 2 types: @{{...}}@ - Javascript computed expression; !{{...}}! - the construction of the translation.

Computed Javascript expressions in tokenized string

This type of tokenization uses a single-line Javascript expression that is placed on a text string. The expression to be evaluated must be placed in the @{{...}}@ construct and the result of its evaluation will be placed in the resulting string.

Example:


let tstring = "Next value: @{{variable + 1}}@";
let result = fcf.tokenize(tstring, {variable: 1});
console.log(result);

Output:


Next value: 2

Variables passed in the second argument of the fcf.tokenize() function, as well as global variables and functions declared in the configuration as available for execution, can be used in the evaluated expression.

Tokenization can be performed both on the browser side and on the NODEJS server side. On the browser side, a eval() call is made to ensure performance, as well as to reduce the size of the sent source JS files. On the NODEJS server side, a simplified execution interpreter is used to execute Javascript instructions, which supports only safe instructions. The interpreter does not use loops, jump statements, or assignment statements. Also, access to predefined global variables and functions that do not write data is allowed. The list of functions available for calling can be found in the configuration file fcf-framework- core:serverConfig.js. For example, to get part of an array, you can use the function slice()

Example:


let result = fcf.tokenize("@{{array.slice(0,2).length}}@", {array: [1,2,3]});
console.log(result);

Output:

But calling the splice method on the server side, which performs array editing, will result in an error.

Example:


try {
  fcf.tokenize("@{{array.splice(0,2).length}}@", {array: [1,2,3]});
} catch(e) {
  console.error(e.toString());
}

Output:


Accessing a field on an invalid element in a command "array.splice(0,2)"

This approach allows you to safely execute calculation expressions that can be supplied from arbitrary sources.

Another distinctive feature of tokenization is the ability to return the original value of the calculation result. To do this, set the result option to "auto" in the third argument of the fcf.tokenize() function. If the tokenized string contains only the @{{JAVASCRIPT_INSTRUCTION}}@ construct, without additional characters, then the fcf.tokenize() function will return the result of the calculation without converting to a string representation.

Example:


let result = fcf.tokenize("@{{variable}}@", {variable: {field1: "value1"}}, {result: "auto"});
console.log(result);

Output:


{ field1: 'value1' }

Translation constructs in tokenized string

Translation substitution constructs !{{...}}!. This type of substitution translates the substring placed in the !{{}}! to the user's language specified in the context (fcf.getContext().language), if no translation is found, the original string will be inserted.

Example:


fcf.getConfiguration().append({
  translations: [
    {
      language: "ja",
      translations: {
        "Hello world": "こんにちは世界",
      }
    }
  ]
});

fcf.getContext().language = "ja";
let result = fcf.tokenize("Text: !{{Hello world}}!");
console.log(result);

Output:


Text: こんにちは世界

Inside the translation construct !{{...}}! it is allowed to use the declaration of the nested design construction @{{...}}@. This approach allows you to do translations for strings with changing data.


fcf.getConfiguration().append({
  translations: [
    {
      language: "ja",
      translations: {
        "Next value: @{{value + 1}}@": "次の値: @{{value + 1}}@",
      }
    }
  ]
});

fcf.getContext().language = "ja";
let result = fcf.tokenize("Text: !{{Next value: @{{value + 1}}@}}!", {value: 1});
console.log(result);

Output:


Text: 次の値: 2

Adding global variables

In a tokenizer, server-side variables are accessed in read-only mode. In addition to the variables passed through the second argument, global variables are also available, only constants from Math and fcf objects are allowed to access global variable data. If you want to add a global variable for access from the tokenizer, then you need to declare it in the configuration.

Example:


(fcf.isServer() ? global : window).globalVariable1 = "value1";
(fcf.isServer() ? global : window).globalVariable2 = { subitem: "value2" };
fcf.getConfiguration().append({
  tokenize: {
    objects: {
      "globalVariable1": "globalVariable1",
      "globalVariable2": "globalVariable2.subitem",
    }
  }
});

let result1 = fcf.tokenize("Global variable 1: @{{globalVariable1}}@");
console.log(result1);

let result2 = fcf.tokenize("Global variable 2: @{{globalVariable2}}@");
console.log(result2);

Output:


Global variable 1: value1
Global variable 2: value2

As shown in the example, adding a variable is done by declaring a property in the tokenize.objects configuration object, where the key is the path to the variable, when accessed from the tokenizer, and the value is a string containing the path to the actual global variable.

Adding functions to the tokenizer

As mentioned earlier, on the server side, when performing tokenization, access to calling functions and methods is allowed only for those functions that do not write data. The list of functions available for calling is predetermined in the framework configuration. This list is declared in the file fcf-framework- core:serverConfig.js. To add the ability to call a function from a tokenizer, you need to add an entry to the tokenize.functions configuration parameter. This configuration parameter is an array containing objects with the following fields:

string|[string] object - the path to the global variable from which the function will be called.
If the variable is equal to the empty string, then the call is made to the global function.

If the last element of the path is equal to the character "*", then permission to call functions on all subpaths is added. For example, if you set the following permission:
fcf.getConfiguration().append({ tokenize: { functions: [ { object: "*", allow: ["toString"], } ] } });
This will allow calling the toString method on any object.

The object property can be set to an array, in which case the first element of the array will be the path used in the tokenizer, and the second element will be a string describing the path to the actual variable.
[string] allow - array of function names allowed to be called
string class = undefined - If the property is set, then the method call is allowed only for objects derived from the given class.

Let's look at a few examples of adding functions to the tokenizer

Example: Example of adding a simple global function


function getString() {
  return "Hello world";
}
(fcf.isServer() ? global : window).getString = getString;

fcf.getConfiguration().append({
  tokenize: {
    functions: [
      {
        object: "",
        allow:  ["getString"],
      },
    ]
  }
});
let result = fcf.tokenize("Value: @{{ getString() }}@");
console.log(result);

Output:


Value: Hello world

In the above example, the function getString() is a global function and therefore the configuration option tokenize.functions[0].object is an empty string.

Example: An example of adding a function located in a global object

If a function is stored in a global object, then the object option must be specified.


(fcf.isServer() ? global : window).Namespace = {
  getString: () => {
    return "Hello world";
  }
};

fcf.getConfiguration().append({
  tokenize: {
    functions: [
      {
        object: "Namespace",
        allow:  ["getString"],
      },
    ]
  }
});

let result = fcf.tokenize("Value: @{{ Namespace.getString() }}@");
console.log(result);

Output:


Value: Hello world

Example: Adding a class with selective resolution of executable methods

To add a class to the tokenizer, you need to add permission to execute the constructor function, to be able to perform the new operation, that is, declare the allow property equal to the value of ["constructor"]. The object option must be set to "*", which will indicate that this permission applies to any path, but specify the class option equal to the class name "Class". Thus, the binding object == "*" && class == "Class", will tell the tokenizer to allow the execution of methods for any object that matches the Class class. The following is an example that illustrates adding a class to a tokenizer with selective resolution of executable methods.


class Class {
  constructor() {
    this.value = 1;
  }

  getString(){
    return "Hello world";
  }
}
(fcf.isServer() ? global : window).Class = Class;

fcf.getConfiguration().append({
  tokenize: {
    functions: [
      {
        "object": "*",
        "class":  "Class",
        "allow":  ["constructor", "getString"],
      },
    ]
  }
});

let propertyValue = fcf.tokenize("Value: @{{(new Class()).value}}@");
console.log(propertyValue);

let methodGetString = fcf.tokenize("getString: @{{object.getString()}}@", {object: new Class()});
console.log(methodGetString);

Output:


Value: 1
getString: Hello world

Example: An example of adding a class with all methods

If all methods can be executed in a class, then full permission can be used for the declaration in the configuration by setting the allow option to ["*"]


class Class {
  constructor() {
    this.value = 1;
  }

  getString(){
    return "Hello world";
  }
}
(fcf.isServer() ? global : window).Class = Class;

fcf.getConfiguration().append({
  tokenize: {
    functions: [
      {
        "object": "*",
        "class":  "Class",
        "allow":  ["*"],
      },
    ]
  }
});

let propertyValue = fcf.tokenize("Value: @{{(new Class()).value}}@");
console.log(propertyValue);

let methodGetString = fcf.tokenize("getString: @{{object.getString()}}@", {object: new Class()});
console.log(methodGetString);

Output:


Value: 1
getString: Hello world

fcf.pattern function

In addition to the fcf.tokenize() function, the framework has a fcf.pattern() function that performs exactly the same tokenization, but the control constructs are @{...}@ and !{...}! (Single curly braces are used). This function ignores the @{{...}}@ and !{{...}}! constructions, so if you need to declare an object in the @{...}@ construction, then you need to add whitespace characters at the beginning or at the end of the control command, so that the tokenizer did not ignore the control command.


let result = fcf.pattern("@{ {value: arg1} }@", {arg1: 1});
console.log(result);

Output:


{ value: 1 }

fcf.inlineExecution function

If you need to safely execute a single-line Javascript command, you can use the fcf.inlineExecution function. This function is similar in execution to a tokenizer, it uses an interpreter on the NODEJS server side, and a eval call on the browser side. The first argument is a JS instruction, and the second argument is an object with additional variables.


let result = fcf.inlineExecution("arg1 ? 'Yes' : 'No'", {arg1: 1});
console.log(result);

Output:

Yes