Issue#86 discover with variables

README.md

@@ -845,6 +845,209 @@ When creating the archive, the tool will try to gather all binaries, scripts, an
 1. Any binaries referenced from the `ORACLE_HOME` will not be gathered, as they are assumed to exist in any target domain to which model-driven operations will be applied.  Doing this is key to allowing the model to be WebLogic Server version independent.
 2. In its current form, the discover domain Tool will only gather binaries and scripts that are accessible from the local machine.  Warnings will be generated for any binaries or scripts that cannot be found but the configuration for those binaries will still be collected, where possible.  It is the user's responsibility to add those missing files to the archive in the appropriate locations and edit the the model, as needed, to point to those files inside the archive using the relative path inside the archive (for example, `wlsdeploy/applications/myapp.ear`).
 
+## The Variable Injector Tool
+
+The variable injector tool is used to tokenize a model with variables. The values for these variables are assigned using an external property file. This facilitates using the same domain model to create new domains in different environments. The variable injector tool can be run as an option in the Discover Domain tool, or or from the stand-alone command line interface.
+
+To enable the Variable Injector during the Discover Domain, create a variable injector configuration by placing a json file named model_variable_injector.json into the \<WLSDEPLOY\>/lib directory using one or more of the pre-defined keywords and/or a CUSTOM list of files. A keyword points to an injector directive file. The tool applies the directives to the attributes in a model, and if the directive matches an attribute, then a property token with a unique variable name is injected into the model and replaces the attribute value. The variable name and model attribute value are placed into the external variable properties file. 
+
+NOTE: Variable injection on an attribute is only performed once. The property token is not replaced by any subsequent matches.
+
+If variable injection is enabled, the Discover Domain Tool calls the variable injector after the model has been discovered and after all filters run, but before model validation.
+
+The supported keywords are as follows:
+
+- CREDENTIALS - All MBean credentials attribute values (user and password) are injected with a variable.
+
+- HOST - All MBean Host attribute values in the model are injected with a variable.
+
+- PORT - All MBean Port attribute values in the model are injected with a variable.
+
+- TARGET - All MBean Target attribute values in the model are injected with a variable.
+
+- TOPOLOGY - Common environmental MBean attributes found in the topology section of the model are injected with a variable. 
+   This includes server, machine and node manager ports, credentials and listen addresses, and cluster messaging modes, addresses and ports.
+
+- URL - All MBean URL attribute values in the model are injected with a variable.
+
+NOTE: The directives used by each pre-defined keyword are defined in an injector json file that is located in the \<WLSDEPLOY\>/lib/injectors folder. These files should not be changed, but could be used as .
+
+Here is an example of a model_variable_injector.json file using the PORT keyword.
+
+```json
+{
+	"PORT": {}
+}
+```
+
+Below is a model snippet that shows injected variables in the port attributes.
+
+```$yaml
+
+topology:
+    Name: soa_domain
+    AdminServerName: AdminServer
+    Cluster:
+        soa_cluster:
+        osb_cluster:
+    Server:
+        AdminServer:
+            ListenAddress: myadmin.example.com
+            ListenPort: @@PROP:Server.AdminServer.ListenPort@@
+            Machine: machine1
+            SSL:
+                Enabled: true
+                ListenPort: @@PROP:Server.SSL.AdminServer.ListenPort@@
+        soa_server1:
+            ListenAddress: managed1.example.com
+            ListenPort: @@PROP:Server.soa_server1.ListenPort@@
+            Cluster: soa_cluster
+            Machine: machine2
+            SSL:
+                Enabled: true
+                ListenPort: @@PROP:Server.SSL.soa_server1.ListenPort@@
+        soa_server2:
+            ListenAddress: managed2.example.com
+            ListenPort: @@PROP:Server.soa_server2.ListenPort@@
+            Cluster: soa_cluster
+            Machine: machine3
+            SSL:
+                Enabled: true
+                ListenPort: @@PROP:Server.SSL.soa_server2.ListenPort@@
+```
+
+And the resulting variable property file:
+
+```
+Server.AdminServer.ListenPort=7001
+Server.AdminServer.SSL.ListenPort=7002
+Server.soa_server1.ListenPort=8001
+Server.soa_server1.SSL.ListenPort=8002
+Server.soa_server2.ListenPort=8001
+Server.soa_server2.SSL.ListenPort=8002
+```
+
+To specify the name and location of the variable properties file for the Discover Tool, use the argument `-variable_properties_file` on the command line. Usage of the variable_properties_file argument without the presence of the model variable injector file in the \<WLSDEPLOY\>/lib directory will cause an error condition and the tool will exit. If the model variable injector file exists in the directory, but the command line argument is not used, the variable properties file is created with the following defaults: If the model_file command line argument is used on the Discover Domain run, the properties file name and location will be the same as the model file, with the file extension `.properties`. If only the archive file argument is present, the archive file name and location will be used.
+
+As with the archive and model file, each run of the discover domain tool will overwrite the contents of an existing variable property file with the values from the current run.
+
+### Custom Variable Injector
+
+To designate custom injector directives, use the CUSTOM keyword in the model_variable_injector.json. The CUSTOM keyword requires a a list of one or more custom injector directive json files. 
+
+An injector directive contains a key that identifies an attribute to be tokenized, and an optional set of directive properties. The key is a period separated MBean hierarchy and attribute name as they are defined in the model. Always exclude the name of the model section from the injector key.
+
+For example, an injector key for the Server SSL Listen Port is as below. This directive contains no additional properties.
+
+```json
+{
+  "Server.SSL.ListenPort": {}
+}
+```
+
+NOTE: The hierarchy of MBeans in the model for the ListenPort attribute and note that the MBean name of AdminServer is NOT included in the directive:
+
+```yaml
+topology:
+    Server:
+        AdminServer:
+            ListenAddress: myadmin.example.com
+            ListenPort: 7001
+            Machine: machine1
+            SSL:
+                Enabled: true
+                ListenPort: 7002
+```
+
+These custom injector json files will be processed by the Variable Injector tool before keywords, each file processed in list order. A property injected into an attribute will not be replaced by any subsequent matches.
+
+#### Custom directive properties
+
+Include the following properties to refine the directive as specified.
+
+- force:<attribute>
+    If the MBean hierarchy exists in the model, but the attribute does not, then the attribute will be added and persisted to the discovered model. The value stored in the
+    model is the weblogic default value.
+
+- variable_value:
+    Replace the model value with the specified value in the variable properties. This may be used in conjunction with the force directive, replacing the default value with the indicated value.
+
+- regexp:
+    A list of regexp patterns that will be applied to either the string values or map values of an attribute in the model. If the pattern matches, then the matching part of the
+    string or dictionary will be injected with a property token and a unique variable name .
+-- pattern:
+    the regular expression pattern to apply to the string value or map values of an attribute
+-- suffix:
+    The suffix name to append to each resulting variable name in order to create a unique variable name
+
+The regexp list is useful when only a segment of a string value or map needs to be tokenized (giving you a clean list of property values in the variable properties file). You can inject more than one token into a string or map with multiple patterns. However, when you have more than one pattern, you must provide a suffix for each. This allows the tool to generate a unique variable name for each token in the string or map.
+
+The following is an example of how to effectively use the regexp directive list to search for a segment in a string value. In this example, we want to search for the host and port in each Oracle JDBC URL that uses the special Oracle URL notation, and create an entry for the host and port in the variable properties file.
+
+In the model, we expect to find a URL like the following:
+
+```$yaml
+    JDBCSystemResource:
+        Database1:
+            JdbcResource:
+                JDBCDriverParams:
+                    URL: 'jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=slc05til.us.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=orcl.us.oracle.com)))'
+```
+
+We create a directive in our custom injector json file:
+
+```json
+  "JDBCSystemResource.JdbcResource.JDBCDriverParams.URL":
+  {
+    "regexp": [
+      {
+        "pattern": "(?<=PORT=)[\\w.-]+(?=\\))",
+        "suffix": "Port"
+      },
+      {
+        "pattern": "(?<=HOST=)[\\w.-]+(?=\\))",
+        "suffix": "Host"
+      }
+    ]
+  },
+```
+
+During the Discover Domain tool run, the pattern is applied to the URL string and tokens injected into the string:
+
+```
+URL: 'jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=@@PROP:JDBCSystemResource.Database1.JdbcResource.JDBCDriverParams.URL--Host@@:)(PORT=@@PROP:JDBCSystemResource.Database1.JdbcResource.JDBCDriverParams.URL--Port@@)))(CONNECT_DATA=(SERVICE_NAME=orcl.us.oracle.com)))'
+```
+
+And The variables put in the properties file:
+
+```
+JDBCSystemResource.Database1.JdbcResource.JDBCDriverParams.URL--Host=slc05til.us.oracle.com
+JDBCSystemResource.Database1.JdbcResource.JDBCDriverParams.URL--Port=1521
+```
+
+#### Selecting specific MBean names for variable injection
+
+This final custom directive allows you to explicitly define which named entries for an MBean in the model you wish to inject properties. For instance, you might wish to tokenize an attribute just for a specific server. To define a list of one or more names for a specific MBean in the injector directive hierarchy, format the list as follows:
+```
+MBean[comma separated list of names]
+``
+To select only the admin server named AdminServer for a Server directive, use the format Server[AdminServer]. To select servers soa_server1 and soa_server2, format the key as Server[soa_server1,soa_server2]
+
+The injector tool recognizes two KEYWORDS for a user list, MANAGED_SERVERS (all the managed servers in the model) and ADMIN_SERVER (The admin server in the model).
+
+A custom injector for the admin server SSL listen port is:
+
+```json
+{
+  "Server[ADMIN_SERVER].SSL.ListenPort": {},
+}
+```
+
+## Samples
+
+A sample of a model_variable_injector.json file and a custom injector json file are installed in the WLSDEPLOY/samples directory.
+
+
 ## Downloading and Installing the Software
 
 The Oracle WebLogic Server Deploy Tooling project repository is located at [`https://github.com/oracle/weblogic-deploy-tooling`](https://github.com/oracle/weblogic-deploy-tooling).  Binary distributions of the `weblogic-deploy.zip` installer can be downloaded from the [GitHub Releases page](https://github.com/oracle/weblogic-deploy-tooling/releases).  To install the software, simply unzip the `weblogic-deploy.zip` installer on a machine that has the desired versions of WebLogic Server installed.  After being unzipped, the software is ready to use, just set the `JAVA_HOME` environment variable to point to a Java 7 or higher JDK  and the shell scripts are ready to run.

core/src/main/python/discover.py

@@ -12,20 +12,20 @@
 from java.lang import IllegalArgumentException
 from java.lang import IllegalStateException
 from java.lang import String
-
 from oracle.weblogic.deploy.aliases import AliasException
 from oracle.weblogic.deploy.discover import DiscoverException
 from oracle.weblogic.deploy.util import CLAException
 from oracle.weblogic.deploy.util import FileUtils
 from oracle.weblogic.deploy.util import PyWLSTException
 from oracle.weblogic.deploy.util import TranslateException
-from oracle.weblogic.deploy.util import WebLogicDeployToolingVersion
 from oracle.weblogic.deploy.util import WLSDeployArchive
 from oracle.weblogic.deploy.util import WLSDeployArchiveIOException
+from oracle.weblogic.deploy.util import WebLogicDeployToolingVersion
 from oracle.weblogic.deploy.validate import ValidateException
 
 sys.path.append(os.path.dirname(os.path.realpath(sys.argv[0])))
 
+import wlsdeploy.tool.util.variable_injector as variable_injector
 from wlsdeploy.aliases.wlst_modes import WlstModes
 from wlsdeploy.exception import exception_helper
 from wlsdeploy.logging.platform_logger import PlatformLogger
@@ -36,8 +36,9 @@
 from wlsdeploy.tool.discover.multi_tenant_discoverer import MultiTenantDiscoverer
 from wlsdeploy.tool.discover.resources_discoverer import ResourcesDiscoverer
 from wlsdeploy.tool.discover.topology_discoverer import TopologyDiscoverer
-from wlsdeploy.tool.validate.validator import Validator
 from wlsdeploy.tool.util import filter_helper
+from wlsdeploy.tool.util.variable_injector import VariableInjector
+from wlsdeploy.tool.validate.validator import Validator
 from wlsdeploy.util import wlst_helper
 from wlsdeploy.util import model_translator
 from wlsdeploy.util.cla_utils import CommandLineArgUtil
@@ -58,8 +59,9 @@
 
 __optional_arguments = [
     # Used by shell script to locate WLST
-    CommandLineArgUtil.DOMAIN_TYPE_SWITCH,
     CommandLineArgUtil.MODEL_FILE_SWITCH,
+    CommandLineArgUtil.DOMAIN_TYPE_SWITCH,
+    CommandLineArgUtil.VARIABLE_PROPERTIES_FILE_SWITCH,
     CommandLineArgUtil.ADMIN_URL_SWITCH,
     CommandLineArgUtil.ADMIN_USER_SWITCH,
     CommandLineArgUtil.ADMIN_PASS_SWITCH
@@ -80,6 +82,7 @@ def __process_args(args):
     __verify_required_args_present(required_arg_map)
     __wlst_mode = __process_online_args(optional_arg_map)
     __process_archive_filename_arg(required_arg_map)
+    __process_variable_filename_arg(optional_arg_map)
 
     combined_arg_map = optional_arg_map.copy()
     combined_arg_map.update(required_arg_map)
@@ -135,7 +138,7 @@ def __process_online_args(optional_arg_map):
                 raise ex
             optional_arg_map[CommandLineArgUtil.ADMIN_PASS_SWITCH] = String(password)
 
-        __logger.info('WLSDPLY-06020')
+        __logger.info('WLSDPLY-06020', class_name=_class_name, method_name=_method_name)
     return mode
 
 
@@ -159,6 +162,28 @@ def __process_archive_filename_arg(required_arg_map):
     return
 
 
+def __process_variable_filename_arg(optional_arg_map):
+    """
+    If the variable filename argument is present, the required model variable injector json file must exist in
+    the WLSDEPLOY lib directory.
+    :param optional_arg_map: containing the variable file name
+    :raises: CLAException: if this argument is present but the model variable injector json does not exist
+    """
+    _method_name = '__process_variable_filename_arg'
+
+    if CommandLineArgUtil.VARIABLE_PROPERTIES_FILE_SWITCH in optional_arg_map:
+        variable_injector_file_name = variable_injector.get_default_variable_injector_file_name()
+        try:
+            FileUtils.validateExistingFile(variable_injector_file_name)
+        except IllegalArgumentException, ie:
+            ex = exception_helper.create_cla_exception('WLSDPLY-06021', optional_arg_map[
+                CommandLineArgUtil.VARIABLE_PROPERTIES_FILE_SWITCH], variable_injector_file_name,
+                                                       ie.getLocalizedMessage(), error=ie)
+            __logger.throwing(ex, class_name=_class_name, method_name=_method_name)
+            raise ex
+    return
+
+
 def __discover(model_context):
     """
     Populate the model from the domain.
@@ -344,7 +369,7 @@ def __persist_model(model, model_context):
             if not model_file.delete():
                 model_file.deleteOnExit()
         except (WLSDeployArchiveIOException, IllegalArgumentException), arch_ex:
-            ex = exception_helper.create_discover_exception('WLSDPLY-06009', model_file.getAbsolutePath(),
+            ex = exception_helper.create_discover_exception('WLSDPLY-20023', model_file.getAbsolutePath(),
                                                             model_file_name, arch_ex.getLocalizedMessage(),
                                                             error=arch_ex)
             __logger.throwing(ex, class_name=_class_name, method_name=_method_name)
@@ -368,25 +393,31 @@ def __check_and_customize_model(model, model_context):
     if filter_helper.apply_filters(model.get_model(), "discover"):
         __logger.info('WLSDPLY-06014', _class_name=_class_name, method_name=_method_name)
 
-
+    inserted, variable_model, variable_file_name = VariableInjector(_program_name, model.get_model(), model_context,
+                                                                    WebLogicHelper(
+                                                                        __logger).get_actual_weblogic_version()).\
+        inject_variables_keyword_file()
+    if inserted:
+        model = Model(variable_model)
     try:
         validator = Validator(model_context, wlst_mode=__wlst_mode)
 
         # no variables are generated by the discover tool
-        validator.validate_in_tool_mode(model.get_model(), variables_file_name=None,
-                                                      archive_file_name=model_context.get_archive_file_name())
+        validator.validate_in_tool_mode(model.get_model(), variables_file_name=variable_file_name,
+                                        archive_file_name=model_context.get_archive_file_name())
     except ValidateException, ex:
         __logger.warning('WLSDPLY-06015', ex.getLocalizedMessage(), class_name=_class_name, method_name=_method_name)
+    return model
 
 
-def __log_and_exit(exit_code, _class_name, _method_name):
+def __log_and_exit(exit_code, class_name, _method_name):
     """
     Helper method to log the exiting message and call sys.exit()
     :param exit_code: the exit code to use
-    :param _class_name: the class name to pass  to the logger
+    :param class_name: the class name to pass  to the logger
     :param _method_name: the method name to pass to the logger
     """
-    __logger.exiting(result=exit_code, class_name=_class_name, method_name=_method_name)
+    __logger.exiting(result=exit_code, class_name=class_name, method_name=_method_name)
     sys.exit(exit_code)
 
 
@@ -433,12 +464,12 @@ def main(args):
                         error=ex, class_name=_class_name, method_name=_method_name)
         __log_and_exit(CommandLineArgUtil.PROG_ERROR_EXIT_CODE, _class_name, _method_name)
 
-    __check_and_customize_model(model, model_context)
-
+    model = __check_and_customize_model(model, model_context)
     try:
         __persist_model(model, model_context)
+
     except TranslateException, ex:
-        __logger.severe('WLSDPLY-06012', _program_name, model_context.get_archive_file_name(), ex.getLocalizedMessage(),
+        __logger.severe('WLSDPLY-20024', _program_name, model_context.get_archive_file_name(), ex.getLocalizedMessage(),
                         error=ex, class_name=_class_name, method_name=_method_name)
         __log_and_exit(CommandLineArgUtil.PROG_ERROR_EXIT_CODE, _class_name, _method_name)
 

core/src/main/python/encrypt.py

@@ -190,7 +190,7 @@ def __encrypt_model_and_variables(model_context):
 
     if variable_change_count > 0:
         try:
-            variable_helper.write_variables(variables, variable_file)
+            variable_helper.write_variables(_program_name, variables, variable_file)
             __logger.info('WLSDPLY-04209', _program_name, variable_change_count, variable_file,
                           class_name=_class_name, method_name=_method_name)
         except VariableException, ve: