Resource Pages for Statistics You Need to Know for Machine Learning

The defining properties or dimensions of big data:

The act of accessing and storing large amounts of information for analytics has been around a long time. But the concept of big data gained momentum in the early 2000s. The mainstream definition of big data includes the three Vs (big data [Laney 2001]):

• Volume: Volume refers to the amount of data from terabytes to petabytes and up. This volume presents the most immediate challenge to conventional IT structures. It calls for scalable storage, and a distributed approach to process such voluminous data.
• Velocity: Velocity refers to the increasing rate at which data flows into an organization and the speed at which such high-velocity data is processed. RFID tags, sensors, and smart meters are driving the need to deal with these torrents of data in near-real time.
• Variety: Variety refers to the number of types of data. Rarely does data present itself in a form perfectly ordered and ready for processing. Data come in all types of formats – from structured, numeric data in traditional databases, to unstructured text documents, emails, videos, audios, stock ticker data, and financial transactions.

At SAS, we consider two additional dimensions when it comes to big data:

• Variability: In addition to the increasing velocities and varieties of data, data flows are unpredictable – changing often and varying greatly. It's challenging, but businesses need to know when something is trending in social media, and how to manage daily, seasonal, and event-triggered peak data loads.
• Veracity: Veracity refers to the quality of data. Because data come from so many different sources, it's difficult to link, match, cleanse, and transform data across systems. Businesses need to connect and correlate relationships, hierarchies, and multiple data linkages. Otherwise, their data can quickly spiral out of control.

The term "big data" refers to data that is so large, fast, or complex that it's difficult or impossible to process using traditional methods. For example, the growth of data created each minute on the internet is as follows. According to the source, Internet live stats, in a single minute, there are:

Demo: Automated Machine Learning in SAS Studio (Self-Paced)

In this demonstration, you access and examine the SAS Studio interface. You then load the course data in memory by using a pre-created SAS Studio flow. By running this flow, you establish a CAS session and then you connect your session to the location of pre-existing data files to load them into memory. Any data processing in SAS Viya requires the data to be loaded in memory first. Finally, you create a flow with the Automated Feature Engineering step to generate features from the PVA_Donors data that can be used as inputs to a predictive model.

Examining the SAS Studio Interface

1. From the Windows taskbar, launch Google Chrome. When the browser opens, select SAS Landing from the bookmarks bar. The user ID and password should be prefilled, but if not, sign in as student using the password Metadata0. (Be sure to use an uppercase M in the password.) If prompted, select Yes for assumable groups.

Note: You can also directly select the SAS Studio page from the bookmarks bar, if available.

2. Click the Applications menu button (the three-by-three grid) in the upper left corner, expand ANALYTICS LIFE CYCLE, and select Develop Code and Flows to launch SAS Studio. The SAS Studio Develop Code and Flows window is displayed. The main window of SAS Studio consists of a navigation pane on the left and a work area on the right. The navigation pane provides easy access to your snippets, steps, and libraries, as well as other components. The work area is used to display your data, code, logs, results, and flows.


3. In the left navigation pane, click the Explorer icon (second from the top). Programs and Flows will be run from the VST folder (Files/Home/workshop/VST). Note: The path to the VST folder may change in later versions of SAS Viya.


4. Next, click the Steps icon (third from the top). Steps can be used to create flows which automate data manipulation and analysis tasks.


5. Now click the Libraries icon (second from bottom) to access the default SAS libraries. You see the WORK, SASHELP, and SASUSER libraries, as well as others. Next, you create and access custom caslibs.

Assigning Libraries and Loading Course Data

Next you run the pre-created SAS Studio flow to assign libraries and load course data. A flow consists of one or more objects. Each object is represented by a node in the process flow. The flow shows the relationship between two or more objects such as a SAS program, a task, a query, and so on. You access pre-created flow as follows:

1. In the left navigation pane, click the Explorer icon (second from top). Expand Files > Home > workshop > VST > AssignLibrary.flw and double-click Assign Library flow.



Note: The SAS Program node in this flow executes a SAS program behind the scenes to establish a CAS session, assign libraries, and load the data in memory. A discussion about the underlying code in the flow is beyond the scope of this course.

2. Run the flow by clicking the Run toolbar button.


3. A green check mark on the Assign Library node indicates that the flow has run successfully. Click the Libraries icon (second from bottom) in the left navigation pane. Look for the VST library and expand it to ensure that the PVA_DONORS table along with other data tables are loaded in memory and are accessible via the VST library. With this step completed, one of the important tasks of loading the data into memory has been accomplished. You analyze this PVA_DONORS data set in upcoming lessons. However, to start with, let's move directly into automated machine learning in SAS Viya.



Note: A pre-made flow with steps for all the demonstrations in this course is located in the VST/backup flows folder.

Optional: After initially running AssignLibrary.flw, this flow is used later to re-establish the VST library whenever SAS Studio times out. As an optional alternative approach, the code from this flow can be added to the Autoexec file. Code in the Autoexec file is executed whenever SAS studio is started. This will result in the VST library being established whenever the user logs into SAS studio. To do this, click on Options at the top left side of SAS Studio, then choose Autoexec file. Copy and paste the code from the AssignLibrary flow to the Autoexec window, then click Save. The VST library will now be available whenever you log into SAS Studio.

Automated Machine Learning

1. In the work area, click the plus sign next to the AssignLibrary.flw tab to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.


2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps tab, expand Data (Input and Output) group, double click on the Table step.


3. In the work area, click on the Table step. At the lower half of the work area, under Table Properties tab, click the file cabinet next to the Library field. Under Libraries on the left, click on the VST library, then select the PVA_DONORS data on the right and click OK.


4. Scroll down through SAS Steps and expand Machine Learning group. Double click on theAutomated Feature Engineering step. The Automated Feature Engineering step explores the variables in the input data and automatically performs feature engineering in a parallel, scalable way.


5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Automated Feature Engineering node (the input port). Now Automated Feature Engineering is connected to the PVA_DONORS data. Click the Arrange nodes button to visually organize the nodes within the flow.


6. Click on the Automated Feature Engineering node. in the lower half of the work area, under the Data tab, find the Target field. Make sure Use a nominal target radio button is selected. Use the plus sign to select a nominal target variable. Scroll down the variable list and click on Response, then choose OK.


7. Under Inputs, ensure that Use all columns except target is selected.


8. Click the Options tab. Keep the settings under Variable Screening, Feature Engineering, and Feature Ranking at their defaults.


9. Click Run to execute the flow. Save the flow to the VST folder as Auto Feature Engineer.flw. When the run is complete, maximize the results window by clicking the three dots on the right side of the work area, choose Open in a browser tab, then choose Results.

Engineered features have been created automatically using appropriate feature transformation and extraction techniques. The idea was to automatically learn a set of features (from potentially noisy, raw data) that can be useful in machine learning tasks without needing manually created, handcrafted engineered features.

In the results window, turn your focus to the Label column. What is so special about this Label column? This is the column that holds the description of the newly generated features.

Note: Other columns in the results window are not discussed here. However, they are covered in detail in other SAS courses.

Going forward, you might want to use these generated features for model building. However, the intention of this demo was to explore the automated machine learning capabilities of the tool.

Bühlmann and Geer (2018) examined at the role of statistics in data science. They focused on issues and challenges beyond computational constraints, namely replicability and its extended concept of stability; heterogeneity in large-scale datasets; causality and its connection to robust prediction; and not beating the uncertainty principle.

Replicability

Replicability means that the findings in a study can be successfully replicated in another study under the same conditions, not exactly but up to statistical error. The statistical error might be quite large, and failure of successful replication might be sometimes not so surprising: a priori, it is difficult to evaluate unless you have an understanding about the statistical uncertainties! A major challenge with big data is to come up with reasonable models, methods, and algorithms that enable you to quantify the statistical error. And you would hope that some findings from big data would indeed enable a fair amount of success for replication in other studies or data sets. Here, replicability should not be confused with reproducibility. Reproducibility, when defined in a narrow technical sense, means that the same data input leads to the same output and numerical results.

Stability

Related to replicability is the notion of stability (Yu, 2013). For example, you can consider the notion of replicability within sub-samples of a large data set. If a finding from half of the data can be successfully replicated in the other half, then the finding seems to be more ''promising'' and ''relevant.'' A methodology for assessing such ''relevance'' is to use repeated splitting of a data set into two halves, and it connects the notion of ''stable findings'' to a Type I error rate in terms of the expected number of false positives. Stability generally turns out to be beneficial for predictive performance as well.

Stability is a useful concept to address whether an alternative, "appropriate" analysis would generate similar knowledge. At the modeling stage, stability refers to acceptable consistency of a data result relative to "reasonable" perturbations of the data or model. For example, jackknife, bootstrap, and cross validation may be considered reasonable or appropriate perturbations if the data are deemed approximately independent and identically distributed based on prior knowledge and an understanding of the data collection process. In addition to modeling stage perturbations, human judgment calls prior to modeling also impact data results. The validity of such decisions relies on implicit stability assumptions that allow data scientists to view their data as an accurate representation of the natural phenomena they originally measured.

Heterogeneity

Stability and replicability usually require some form of homogeneity and independence. In the simplest terms, the first and second data set (or the first and second half of a single data set) are independent and of the same nature. Big data are often believed to be of heterogeneous nature, violating such a homogeneity assumption. In other words, large-scale data, terminology that we use in the sequel for data that have both large sample size and dimensionality, is typically not generated as i.i.d. or stationary samples from a single population distribution. And we have good reasons to expect that the data is heterogeneous, arising from different ''scenarios,'' ''regimes,'' or ''sub-populations.'' Quantifying the statistical error and the amount of replicability that we can expect under such circumstances are important for drawing generalizing and scientific conclusions from large-scale data.

An established way to assign uncertainties and the amount of replicability is to divide this large-scale heterogeneous data into smaller homogeneous sub-data sets, and these sub-data sets are the different studies from which one pursues an aggregation of significance statements. This is a straightforward concept and computationally attractive because computing for every environment can be trivially distributed.

Causality

Heterogeneity in large-scale data seems to be an unpleasant obstacle. However, it can be advantageously exploited for causal inference and robustness against worst case scenarios. There is an interesting invariance assumption, related to stability, that says there exists a subset of covariates such that the conditional distribution of response given inputs is invariant across all environments. Consider the difference between a prediction that is made with a causal model and that with a non-causal model.

Suppose you intervene on the input variables or change the whole environment. The predictions from a causal model will, in general, work as well under interventions as for observational data. In contrast, predictions from a non-causal model can potentially be very wrong if you actively intervene on input variables. We can exploit this invariance of a prediction under a causal model for causal inference. Given different experimental settings, we collect all models that do show invariance in their predictive accuracy across settings and interventions. The causal model will be a member of this set of models with high probability. This approach yields valid confidence intervals for the causal relationships in quite general scenarios.

Uncertainty

Uncertainty can be mathematically well understood with little practical consequence because this uncertainty is not something you can directly see from data. One needs another algorithm to access uncertainty, and this algorithm has its own uncertainty. Statistics might quantify how far we can go (for example, a classical statistical uncertainty measure is the p-value). An important task is to validate machine learning or statistical algorithms (for example, their predictive performance or their ability to find important or even causal variables). Validation of prediction is certainly easier than assessing the performance to find important or causal variables.

However, already the task of prediction is not trivial at all. Standard cross validation is inappropriate for heterogeneous data, and assessing the prediction performance for new unseen scenarios is highly complex. The modern adaptive algorithms are often such that the better they are (generate outcomes close to the target), the harder it is to estimate their uncertainty. For the ''best'' algorithm, it can be simply impossible to know its accuracy. To quote Marc Hoffmann and Oleg Lepski: ''You know adaptive estimators converge very fast if the function is very smooth (or has a prescribed complexity) but you can tell nothing about the estimated function itself.'' (Hoffmann and Lepski, 2002). The statistical theory serves as a guard against cheating with data. You cannot beat the uncertainty principle.

At the heart of SAS Viya, is SAS Cloud Analytic Services, or CAS. CAS is a high-performance server that provides the run-time environment for data management and analytics.

Many of the tools that SAS offers to facilitate parallel task execution rely on the asynchronous action execution framework provided by CAS. Others rely on distributing calculations to individual worker nodes. Within CAS, there are tools that handle parallel execution automatically for you, as most CAS actions employ some form of parallel processing. There are also tools that enable you to write your own parallel task execution frameworks.

CAS Sessions

When you sign in to SAS Studio in SAS Viya, SAS creates a single session with the workspace server. You use the workspace server as you did in SAS 9 or in other prior SAS releases. To use CAS, you explicitly start a session with the CAS server.

When you create a session, the server first authenticates your identity. If your identity is successfully authenticated, the server starts the session controller process. That process sends a message to each machine in the cluster to start a session process for your client.

You use the CAS statement to create a session. Add the CAS statement to your program either by entering the statement or by using the SAS Studio code snippet New CAS Session.

CAS is designed to run in a single-machine symmetric multiprocessing (SMP) or multi-machine massively parallel processing (MPP) configuration, supporting multiple platform and infrastructure configurations.

In the single-machine configuration, all processing is completed on one node. The multi-machine configuration, however, consists of one controller and one or more worker nodes, and it provides optimal processing capabilities.

When performing analytics, CAS efficiently spreads big data processing across all nodes in the cluster, which results in very fast operations.

In addition, CAS has a communications layer that supports fault tolerance, meaning it can continue processing requests even after losing connectivity to some nodes. It does this by distributing work to other nodes. This communications layer also enables you to dynamically delete and add nodes to the system while the server is running.

SAS Viya Architecture

Here are some of the basic areas where SAS 9 and SAS Viya differ:

Front Ends

In the SAS 9 environment, there are different front-end technologies. This includes front ends based on Java such as SAS Management Console and SAS Data Integration Studio. Other front ends such as SAS Enterprise Guide and SAS Add-In for Microsoft Office are in C (.Net) based desktop applications. In addition, there are also browser-based web applications in the SAS 9 environment. The front ends for SAS Viya have been developed from scratch as pure browser-based applications based on HTML5. There are no "fat client" desktop applications.

Processing

The processing of data in a SAS 9 environment is typically single threaded and sequential in a SAS process after reading the data from the file system or from a DBMS. Some of the SAS procedures are multi-threaded and thus achieve a significant reduction in run time. In addition to the processing of the data by the processor (or processors), the processing speed is influenced by the reading/writing of data to/from file systems.

SAS Viya is based on the in-memory engine of SAS Cloud Analytic Services and has been developed and optimized from processing to distributed processing of processes in a cluster of computers. SAS Viya action sets and actions are multi-threaded and, in combination, with the distributed processing in the in-memory engine, allow a significant reduction in processing time when processing large amounts of data.

Accessing Data and Data Sources

In a SAS 9 environment, the (pre)processing of data can be executed based on SQL in a DBMS. However, the actual execution of SAS 9 DATA and PROC steps requires the data in the SAS process. With SAS Viya, processing can be moved directly to data storage using SAS In-Database technology (SAS Data Connect Accelerator and also SAS Embedded Process). Transporting data from storage to SAS Viya processing is not always necessary. In addition to this, SAS Data Connectors exist in the SAS Viya environment to access other data sources.

Architecture: Single Machine

In a single-machine architecture, the Cloud Analytic Services controller services requests and acts as the controller. The controller node performs data analysis on the rows of data that are in memory. A SAS Data Connector is used to configure connections to data sources such as existing SAS data, Teradata databases, and Hive data in Hadoop.

The single machine uses the available CPUs and threads of the system to speed up data analysis. All the in-memory analytic features of a distributed server are available to the single-machine server. Single-machine servers cannot load data into memory in parallel. CAS Server Monitor is available only in a programming-only deployment.

Architecture: Distributed Environment

The distributed server consists of one controller, an optional backup controller, and one or more workers.
When a CAS server is running in massively parallel processing (MPP) mode, in addition to a controller, the CAS server also assigns multiple machines as CAS workers. The controller parcels out work to each worker node. The worker nodes perform coordinated parallel processing of the distributed data and generate results that are sent back to the controller. Each of the CAS nodes has a SAS Data Connector that provides connectivity to data sources. In addition, the SAS Data Connect Accelerator allows for embedded processes to run in data sources like Hadoop.

CAS has a communications layer that supports fault tolerance, so when CAS is running in an MPP configuration, it can continue processing requests even after losing connectivity to some nodes. This communication layer also enables you to remove or add nodes while the server is running. Also, if the CAS controller machine goes down, you can configure an optional backup controller in an MPP environment.

For both architectures, the server is multi-threaded for high-performance analytics.

SAS Cloud Analytic Services (CAS)

One of the supporting services of SAS Viya is SAS Cloud Analytic Services, or CAS. CAS is an in-memory, distributed, analytics engine. It uses scalable, high-performance, multi-threaded algorithms to rapidly perform analytical processing on in-memory data of any size.

SAS Viya is open. The word "open" signifies the fact that the power of SAS to build and deploy analytics can be accessed via many programming languages—not just SAS, but also Python, R, Lua, Java, or RESTful APIs. This integration enables analytical teams with varied backgrounds and experiences to come together and solve complex problems in new ways. Integrating SAS and open-source technologies is often advantageous in two main scenarios: programmatically accessing the SAS Platform using open-source software and bringing open-source models into the SAS Platform for side-by-side comparison.

SAS Viya contains microservices. A microservice is a small service that runs in its own process and communicates with a lightweight mechanism (HTTP). Microservices are a series of containers that define all the different analytic life cycle functions (sometimes described as actions) that fit together in a modular way. The in-memory engine is independent from the microservices and allows for independent scalability.

On the left side of the SAS Viya architecture graphic above, you see a series of source-based data engines.

SAS Viya has a middle tier implemented on a microservices architecture, deployed and orchestrated through the industry standard cloud Platform as a Service, Cloud Foundry. Through Cloud Foundry, SAS Viya can be deployed, managed, monitored, scaled, and updated. Cloud Foundry enables SAS Viya to support multiple-cloud infrastructure, enabling customers to deploy SAS in a hybrid cloud environment that spans multiple clouds, including the combination of on-premises cloud infrastructure and public cloud infrastructure.

You can choose to use other platforms such as Docker and the open container initiative. You can operate on a private infrastructure such as openstack and vmware or an open infrastructure such as Amazon Web Services and Azure.

Existing SAS solutions and new ones are being built in SAS Viya. In addition, you can use REST APIs to include SAS Viya actions into your existing applications.

What have you learned?

• There are different types of data: experimental, opportunistic, and big data.
  
  
• Each type of data is analyzed differently, using statistics, data mining, and machine learning.
  
  
• Machine learning and deep learning are subfields of artificial intelligence, and deep learning is a specific field of machine learning.
  
  
• Big data produced smart applications, such as computer vision, natural language processing, and internet of things.
  
  
• Machine learning uses many statistics and data mining concepts and methods.
  
  
• In the use of statistical modeling to reach conclusions from data, there are two cultures: the data modeling culture and the algorithmic modeling culture.
  
  
• Sampling might or might not be needed for modeling big data (SEMMA > EMMA, EMSMA, sEMMA, sEMSMA).
  
  
• There are equivalent machine learning terms for most of the statistical terminology, and they all point to the same statistical concept.
  
  
• Variables come in two different types: categorical and continuous.
  
  
• You must determine the type and level of measurement for your variables: binary, nominal, ordinal, interval, and ratio.
  
  
• Big data demands sophisticated data management and advanced analytics techniques -- SAS Viya.
  
  
• SAS Studio is an interactive development environment to jump start your data and analytics efforts.

Probability Sampling

In probability sampling, each element in the population is given a known probability of selection. A random mechanism is used to select a specific number of objects for the sample. For example, 12 objects are randomly selected out of 36, so that every object gets an equal and fair chance of selection. Probability sampling can help ensure that the sample is representative of the population.

Nonprobability Sampling

Sampling techniques used to select objects from the population by any mechanism that does not involve a random selection process is called nonprobability sampling. In nonprobability sampling, not every object of the population has a chance of being included in the sample. For example, 12 objects are selected in a square pattern because you like the square shape. Because chance is not used to select these objects for the samples, these techniques are uncommon in data mining and machine learning world and therefore not discussed in this course.

The result of undersampling is a sample with fewer examples in the majority class as vast quantities of data are discarded. This can be problematic because the loss of such data can make the classifier harder to learn, resulting in a loss in classification performance. Undersampling is effective in situations in which the minority class has enough examples despite the severe imbalance.

The advantage of undersampling is that, on average, you can obtain a model of similar predictive power with a smaller overall case count. This is in concordance with the idea that the amount of information in a data set with a categorical outcome is determined not by the total number of cases in the data set itself, but instead by the number of cases in the rarest outcome category. For binary target data sets, this is usually the event (Harrell 2006). Undersampling significantly reduces analysis time. This advantage might seem of minimal importance in the age of extremely fast computers. For example, a model might fit 10 times faster with a reduced data set, but a 10-second model fit versus a 1-second model fit is probably not relevant. However, the model-fitting process occurs only after the completion of a long, tedious, and error-prone data preparation process. Smaller sample sizes for data preparation are usually welcome.

For machine learning algorithms affected by skewed distribution, such as neural networks and SVMs, oversampling is a highly effective technique. However, it might increase the likelihood of overfitting the minority class because it makes exact copies of the minority class examples, in turn resulting in an increase of generalization error. Another thing you must be aware of is the increased computational cost.

Demo: Listing Observations (Self-Paced)

In the previous lesson, you loaded all the course data into memory. In this demonstration, you use a SAS Studio flow to list the first 10 observations in the PVA_DONORS data set.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.


2. In the navigation panel on the far left, click the Explorer button. Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.


3. Click the Run button on the flow toolbar.

Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.


2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.


3. In the work area, click on the Table step. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK.


4. Under SAS Steps, expand Examine Data, then double click on List Data.


5. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the List Data node (the input port).Now List Data is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


6. Click on the List Data node, then click on theOptions tab. Scroll down to Rows to list field and use the dropdown arrow to select First n rows. Leave the default value of Number of rows set to 10 rows.


7. Use the save button to save the flow to the VST folder as Explore PVA.flw. Click Run to run the flow. To view the results, click on Submitted Code and Results, then click Results.

Alternatively, click the three vertical dots on the upper right corner of the open tab, then choose Open in a browser tab, then Results, to open the results in a new browser tab.

• Use the double arrows in the upper right of the navigation pane to hide the navigation pane and maximize the results. The results appear in the Work area. To increase the size of the work area, locate the List Data step settings in the lower half of the work area. Click the Minimize preview button (a minus sign icon) at the top right corner of this section.


• The successful submission of the List Data step produces a list of the first 10 observations from the PVA_DONORS data set. Listing data provides a quick snapshot of your analysis table. By default, the output of the List Data step shows variable labels. You can easily identify the gift-related inputs, such as Gift Count 36 Months, Gift Amount Last, and so on, that display the previous donation history. Scrolling to the right, you can see the demographic inputs, such as Demographic Cluster, Age, and Gender, that provide demographic characteristics about the donors. And last but not the least, there are the two target variables, Target Gift Flag (Response) and Target Gift Amount (Donation_Amt). The binary variable Target Gift Flag represents the response to the campaign and has two levels, Yes and No. The continuous variable Target Gift Amount represents the donation amount from the individuals who have responded to the campaign.


• You will use both target variables in this course, but only one at a time.

One definition of outliers is data that are more than 1.5 times the interquartile range before Q1 or after Q3. Why "1.5"?

Because the quartiles for the standard normal distribution are +/-.67, the IQR = 1.34. Thus, 1.5 times 1.34 = 2.01, and outliers are less than -2.68 or greater than 2.68. Therefore, for normally distributed data, the probability of being an outlier is 2 times .0037 = .0074. This is less than 1%. Thus, the 1.5 IQR method results in an outlier being data that is more extreme than 99% of the data values for a normally distributed variable.

A negative kurtosis statistic means that the data has lighter tails than in a normal distribution. Data is less heavily concentrated about the mean. You refer to this type of distribution as platykurtic. This type of distribution is flat and spread out. In this example, the platykurtic distribution is symmetric. A symmetric platykurtic distribution is characterized by being flatter than the normal distribution, that is, less peaked, with heavier flanks and thinner tails. A distribution with two peaks is also referred to as a bimodal distribution. Rectangular, bimodal, and multimodal distributions tend to have low values of kurtosis.

A positive kurtosis statistic means that the data has heavier tails and is more concentrated about the mean than a normal distribution. You refer to this type of distribution as leptokurtic. This type of distribution is high and thin. A leptokurtic distribution is often referred to as heavy-tailed and might sometimes also be referred to as an outlier-prone distribution. If the distribution is symmetric, a leptokurtic distribution tends to have a higher peak than the normal, with an excess of values near the mean and in the tails, but with thinner flanks. Distributions that are asymmetric also tend to have nonzero kurtosis. In these cases, understanding kurtosis is considerably more complex than in situations where the distribution is approximately symmetric. Kurtosis can be difficult to assess visually.

Demo: Exploring Data (Self-Paced)

This demonstration illustrates performing initial data exploration in three parts.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries,LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.


2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.


3. Click the Run button on the flow toolbar.

Part 1: Obtaining Summary Statistics for Continuous Variables

In this demonstration, you perform initial data exploration using a flow with the Summary Statistics node. You obtain summary statistics for continuous variables.

1. Open the previously saved flow Explore PVA.flw from the VST folder.


2. Under SAS Steps, expand the Statistics group, then double click on the Summary Statistics step.


3. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Summary Statistics node (the input port). Now Summary Statistics node is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


4. Click on the Summary Statistics node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Click the box next to Name to select all the variables. Click OK.


5. Click on the Option tab. The Mean, Standard deviation, Minimum value, Maximum value, and Number of observations are selected by default. Keep these options and additionally select Number of missing values.


6. Still under the options tab, expand Additional Statistics. Select Mode, Skewness, and Kurtosis. Expand Percentiles, then check off Lower quartile, Median, and Upper quartile.


7. Save the flow then right-click on the Summary Statistics node and choose Run node. Open the Results in a new browser tab.

• Notice that several observations have missing entries for the variables GiftAvgCard36 (Gift Amount Average Card 36 Months), DemAge (Age), and Donation_Amt (Target Gift Amount). The Donation_Amt variable has as many as 50% (4843 out of 9686) of its values missing. Also, skewness statistics indicate that most of the variables with the prefix Gift are positively skewed with skewness values ranging from approximately 1.1 to 14.


• Then you examine the DemMedIncome (Median Income Region) variable. Note that it has a minimum value of zero and a very large standard deviation (28707.49). This looks suspicious and warrants further scrutiny. A closer look at its mode (also zero), which is significantly different from its mean and median, reveals an important insight. That is, out of all the income values, zero is the most frequent. It appears that these are the cases for which an income is unknown and has been coded as zero (0). Clearly, having zero income is considerably different from having an unknown income.


• If you properly use the income variable in a predictive model, the solution to this situation is to recode these values as missing and then impute. This solution is presented in upcoming lessons.


• Notice the StatusCatStarAll (Status Category Star All Months) variable, which has a minimum value of zero and a maximum value of 1. In fact, it's a binary variable. By default, SAS treats every numeric variable as interval and every character variable as nominal. Going forward, we will ensure that we use this variable as a categorical variable rather than an interval variable.


• Among the other variables, let's examine the summary statistics of the interval-valued target variable Donation_Amt (Target Gift Amount). You see that the mean (101.64), median (102.6), and mode (92.10) are not very different from one another, with a skewness value (-0.086) being close to zero. These statistics give us the impression that the distribution of the Donation_Amt variable is fairly symmetric. However, it's always recommended to examine the distribution plots of variables to graphically assess the shape and spread of the data. Data satisfying normal distribution is beneficial for model building and is required in upcoming lessons on regression modeling. Next, you examine the distribution of continuous variables.

Part 2: Distribution Analysis of Continuous Variables

In this demonstration, you use a flow with the Distribution Analysis node to examine the distribution of continuous variables.

1. Return to the previous flow containing the Summary Statistics step (Explore PVA.flw). From the Steps menu, under SAS Steps, expand Statistics, then double click on Distribution Analysis.


2. Move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Distribution Analysis node (the input port). Now Distribution Analysis is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


3. Click on the Distribution Analysis node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Add all the variables except for StatusCatStarAll. A quick way of getting this done is by clicking the select all box next to Name and then deselecting StatusCatStarAll. Click OK to close the Column Selection window.


4. Click the Options tab. Verify that Histogram has been selected. Select the Add normal curve box under Exploring Data to include a normal curve for the histogram. To include specific summary statistics, select the Add inset statistics option and expand the Inset Statistics properties and select Mean, Standard deviation, Skewness, and Kurtosis in addition to the default of Number of observations. To assess normality using the normal probability plot, select the Normal probability plot box under Checking for Normality.


5. Right click on the Distribution Analysis node and choose Run node. Save the file Explore PVA.flw. When the run is complete, you can view the results on a browser tab. Click the More options on the right side of the screen (at the top right side of the open tab). Click Open in a browser tab and then click Results. View the results.

Note: If you see a warning message in the results window that says The maximum size limit has been exceeded, you might want to click the Open in new tab link to view the results. Alternatively, go to the SAS Studio Options menu, and select Preferences. Under SAS Programs, click Results. Under Display warning if results are larger than specific size, change the Maximum result size (MB) from 3 to 10.

Your results window displays histograms for all the analysis variables assigned in the Analysis variables property. In a practical scenario, it is worth spending time to examine the shape and spread of all the analysis variables in data. However, this is overwhelming in the class setup. So, let's examine the distribution of a few chosen variables. Consider the histogram for the Gift Count 36 Months variable.

The histogram and the normal curve superimposed on it indicate that the distribution is positively skewed, and the same can be ascertained from the statistics in the inset box. Scroll further down to examine the distribution of the DemMedIncome variable.

You clearly see a spike around the zero value, which accounts for approximately 25% of the income values. This resonates with the summary statistics, which indicate its mode value to be zero.

Now you consider examining the distribution of our interval-valued target variable: Donation_Amt (Target gift amount).

The histogram superimposed with the normal curve indicates that the variable is relatively normally distributed.

Next, to examine the normal probability plot for Donation_Amt, scroll down further until the bottom of page, and you see that the distribution of data points follows the normal reference line quite closely.

Thus, the data distribution looks fairly normal. This is an important insight about the interval valued target variable which could be useful in upcoming lessons on modeling.

Part 3: Generating Tables and Plots for Categorical Variables

In this demonstration, you use a flow with the One-Way Frequencies node to generate frequency tables and frequency plots from your data.

1. Return to the previous flow Explore PVA.flw. From the Steps menu, under SAS Steps, expand Statistics, then double click on One-Way Frequencies.


2. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the One-Way Frequencies node (the input port). Now One-Way Frequencies is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.


3. Click on the One-Way Frequencies node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Add the variables StatusCat96NK, StatusCatStarAll, DemCluster, DemGender, DemHomeOwner, and Response variables.


4. No changes are needed in the Options tab.


5. Save the flow. Click right click on the One-Way Frequencies node and click Run node. Click on Submitted Code and Results, then click Results to view the report. Alternatively, click the More options on the right side of the screen (three vertical dots at the top right side of the open tab), then click Open in a browser tab, and finally click Results.

The flow displays the frequency table and bar charts for the requested variables. Note that the DemCluster variable has as many as 54 distinct levels. Do you see any challenges involving using a categorical variable that has as many as 54 levels as input for model building? Scroll further to examine the distribution of yet another target variable in our data set, Response. The frequency table and distribution plot clearly indicate that the data set has an equal proportion of responses and non-responses.

Do you see any relationship between proportion of non-responses (in the Response variable) and the number of missing values in Donation_Amt?

Confidence intervals are useful for communicating the variation around a point estimate.

If you want to calculate a confidence interval, you need four values: (1) the point estimate you are constructing the confidence interval for, (2) the critical values for the test statistic, (3) the standard deviation of the sample, and (4) the sample size. As an example, here's the formula for calculating the confidence interval for the mean:

$\begin{array}{l}\overline{x}\pm t\cdot s_{\overline{x}}\\ s_{\overline{x}}=\frac{s}{\sqrt{n}}\end{array}$

$\overline{x}$ is the sample mean.

$t$ is the $t$ quantile value that is determined by the confidence level and the sample size.

$s_{\overline{x}}$ is the standard error of the mean.

The t-value that you will use is dependent on the confidence level that you choose. Confidence, in statistics, is another way to describe probability. Your desired confidence level is usually one minus the alpha (α) value you would generally use in your statistical hypothesis test. The value after the $\pm$ symbol is known as the margin of error.

Demo: Testing a Hypothesis Using a One-Sample t-Test (Self-Paced)

In this demonstration, you use a flow with the t-Tests node to test whether the population mean donation amount is $100 given the information in the PVA_DONORS table.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.
3. In the work area, click on the Table step. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK. Note that it is also possible to copy the PVA_DONORS node from a previously created flow and paste it into the new flow.
4. Under SAS Steps, expand Statistics, then double click on t-Tests.
5. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the t-Tests node (the input port). Now t-Tests is connected to the PVA_Donors data.
6. Click on the t-Tests node, then select the Data tab. Verify that One-sample test is selected under the t-test field.
7. Click the plus sign for the Analysis variable field. Assign Donation_Amt as the analysis variable. Click OK to close the Column Selection window.
8. On the Options tab, in the Alternative hypothesis: mu ≠ field, enter 100.
9. Under Normality Assumption, clear the option to conduct Tests for normality.
10. To select plots, expand the Plots property and use the drop-down menu to choose the option Selected plots. Select the option to display a Confidence interval plot in addition to the default plots of histogram, a box plot, and a normality plot.
11. Save the flow to the VST folder as ttest.flw.
12. Run the flow and examine the results.

The mean donation amount is $101.60, and the associated t statistic value is 4.44. The p-value is <0.0001. Assuming a significance level of 5% (that is, alpha=0.05), you find that the p-value is less than alpha. Thus, you reject the null hypothesis and conclude that the hypothesized population mean donation amount of $100 is significantly different from the sample mean of $101.60. Scroll further to examine the confidence interval plot. It is a horizontal line ranging from 100.9 to 102.4 to graphically represent the 95% confidence interval around the mean estimate of the donation amount.





Notice the vertical reference line drawn at 100 (the hypothesized value). This reference line lies outside the confidence limit, which clearly indicates that the mean value in the sample is statistically different from $100 at an alpha level of 0.05.

Next, the histogram and the q-q plot for donation amount do not show any severe departures from normality although there are some potential outliers present in the data. Therefore, the Student's t test is valid.

At this point, your brain might be struggling to accept whether a difference of $1.60 between the hypothesized value and the sample mean is significant when the Donation_Amt variable ranges from $0 to $211.9. Well, the interpretation that you just made from the t-test is based on the data in hand and the significance level chosen. Recall our discussion on the over-reliance on the arbitrary cutoff of a 5% significance level and the sample size. In inferential statistics the 5% level of significance might work well given the sample size. However, in a big data context, it is not always appropriate. Large sample sizes result in smaller p-values, and in that case, you tend to reject the null hypothesis almost every time. Thus, you better select a very small significance level cutoff when using the p-value approach in the big data world. In this example, we have a significantly large data set (from a statistical perspective) with 4843 observations. Thus, using a 5% level of significance is highly likely to produce some statistically significant results.

Let's see the impact of using a smaller significance level cutoff on the inference that we draw and the confidence intervals.

Changing the Values of the Level of Significance in the t-Test

13. Navigate to the VST course folder and double click on the SAS program file Ttest change alpha.sas.
14. Look at the SAS program to see the specified alpha level for this ttest:






proc ttest data=VST.PVA_DONORS sides=2 h0=100 plots(only showh0)=
	(summaryPlot intervalPlot qqplot) alpha=0.000001;
   var Donation_Amt;
run;



15. Click the + Code to Flow button and add the program to the previously saved ttest flow.



16. Return to ttest.flw. Right click on the ttest change alpha node and click Run node. Save the flow and examine the Results.
    
    
    

    Note that neither the t statistic value nor the corresponding p-values have changed in comparison to previous run results. What changes is the confidence interval for the mean donation amount and the statistical inference.

    
    
    
    
    

    Now you see that the 99.9999% confidence limit values for the mean range from 99.83 to 103.4, which clearly includes the hypothesized value within it. The same interpretation can be drawn from the confidence intervals plot. You see that the reference line at $100 is contained within the confidence limit. Thus, you would reach the conclusion that the mean donation amount is not statistically different from $100.

    Note that there is nothing special about considering a significance level of 5% always. It should depend on the problem that you are studying and the sample size.

    
    

What have you learned?

• The study of statistics can be divided into descriptive statistics and inferential statistics.
  
  
• Population parameters are generally unknown and can be estimated or tested using sample statistics.
  
  
• Sampling is broadly classified into probability and non-probability sampling.
  
  
• Event-based sampling is extremely important in machine learning predictive models -- oversampling and undersampling.
  
  
• The PVA _Donors data has more than 9,000 rows and 29 columns. There is one binary target and one (related) interval target.
  
  
• Using measures of shape, like skewness and kurtosis, in conjunction with measures of central tendency and dispersion, makes possible a near complete numerical description of your data.
  
  
• In addition to plotting your data using histograms, you can use box plots and normal probability plots to compare your data with the normal distribution.
  
  
• Normal distribution assumption is useful in many machine learning tasks, including data preparation.
  
  
• You often consider values that are more than two or three standard deviations from the mean as unusual.
  
  
• Higher confidence levels result in wider intervals.
  
  
• In machine learning, the term inference is sometimes used to mean "make a prediction, by evaluating an already trained model".
  
  
• The p-value is the probability of obtaining a test statistic as extreme or more extreme than the one observed in your data given that the null hypothesis is true.
  
  
• A one-sample t test compares the mean calculated from a sample to a hypothesized mean.
  
  
• Large sample sizes result in smaller p-values that often leads to reject the null hypothesis. With big data, a significance level of .01 or smaller is suggested.

Another method for quantifying linear relationships is linear regression. The term regression was first used in the 1880s by Sir Francis Galton.

Galton observed that parents tended to have children who grew up to heights that were closer to the average than their own heights. In other words, when parents were taller than average, their adult children tended to be shorter than their parents. And when parents were shorter than average, their adult children tended to be taller than their parents. He called this pattern regression towards mediocrity, with mediocrity referring to the average or mean.

The modern meaning of the word regression is a statistical tool that enables us to describe the relationship between two or more continuous variables. The relationship is estimated by expressing one variable (the dependent variable Y) as a linear function of the one or several other variables (the independent variables, or Xs). 

$Y={\beta }_0+{\beta }_1+\epsilon$

Simple regression equation:

$\widehat{Y}={\widehat{\beta }}_0+{\widehat{\beta }}_{1}X$

The method of least squares produces parameter estimates with certain optimum properties. Estimated parameters are denoted with a hat above the parameter (in this case, ${\widehat{\beta }}_0$ and ${\widehat{\beta }}_1$). If the assumptions of simple linear regression are valid, ${\widehat{\beta }}_0$ and ${\widehat{\beta }}_1$ are unbiased estimates of the population parameters and have minimum variance.

The least squares estimators are often called BLUE (best linear unbiased estimators). The term best is used because of the minimum variance property. Unbiased means that when the parameters are estimated many time, the average estimate is equal to the true population parameter. Because of these optimal properties, the method of least squares is used by many data analysts to investigate the relationship between continuous predictor and response variables.

Many of the mathematical details of linear regression can be summarized in an ANOVA table. ANOVA is short for Analysis of Variance, a statistical tool usually associated with partitioning variability among levels of categorical predictors. The ANOVA table is also used in linear regression. It summarizes the model and error degrees of freedom, sums of squares, and mean squares. The mean squares for the model and the error are used to construct an F-ratio, a statistic used along with the degrees of freedom to test the significance of the regression model. The significance is assessed through a p-value. If the p-value is lower than the significance threshold alpha (not shown), the model is significant, meaning at least one predictor is statistically different from zero.

The model row of the ANOVA table corresponds to the variance that can be explained by the regression equation. The error row of the ANOVA table corresponds to the unexplained variability. The error is a result of both true randomness as well as any important predictors not included in the model.

As for the parameter estimates, the slope and intercept regression parameters can be estimated using the pictured equations.

Estimated slope: ${\widehat{\beta }}_1=\frac{\sum \left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{{\left(x_i-\overline{x}\right)}^2}$

Estimated intercept: ${\widehat{\beta }}_0=\overline{y}-{\widehat{\beta }}_1\overline{x}$

For a simple linear regression, the slope equation can be described as the sums of XY cross-products divided by the sums of squares for X. Once the slope is calculated, the intercept can be calculated as the mean Y minus the slope times the mean of X.

Demo: Correlation and Linear Regression (Self-Paced)

In this demonstration, you explore relationships between numeric predictors and the response variable Donation_Amt using correlation analysis in a SAS Studio flow. Then the input GiftAvgCard36, which is most correlated with the target, is used in a simple linear regression.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.


2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.


3. Click the Run button on the flow toolbar.

Demo Steps

Correlation analysis

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.


2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps tab, expand Data (Input and Output) group, double click on the Table step.


3. In the upper half of the work area, click on the Table step. In the lower half of the work area, under Table Properties tab, click the file cabinet next to the Library field. Under Libraries on the Left, click on the VST library, then select the PVA_DONORS data on the right and click OK.


4. Scroll down through SAS Steps, expand Statistics group, then double click on Correlation Analysis.


5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Correlation Analysis node (the input port). Now Correlation Analysis is connected to the PVA_DONORS data. Click the Arrange nodes button to visually organize the nodes within the flow.


6. Click on the Correlation Analysis node. In the lower half of the work area, under the Data tab, find the Analysis variables field and click the plus sign to add variables. Add all variables except Donation_Amt and StatusCatStarAll. Do this by clicking the Select all box next to Name and then deselecting Donation_Amt and StatusCatStarAll. Click OK.


7. Find the Correlate with field. Use the plus sign to add the variable Donation_Amt. Click OK.


8. Switch tabs to Options. Under Statistics > Statistics to display, select Selected statistics from the drop-down menu. Check the boxes for Display p-values and Order correlations from highest to lowest (in absolute value).


9. Scroll down and expand Plots. Under Type of plot, select Individual scatter plots from the drop-down menu. Check the Include ellipse box and use the default settings of Prediction Ellipse for the ellipse and 95% for the confidence level.


10. Uncheck the Include inset statistics box.


11. Keep Number of Variables to plot set to 5 and change Maximum number of plot points to No limit.


12. Click Run to execute the flow. Click on the Submitted Code and Results tab at the top. The results appear in the work area. To increase the size of the work area, locate the Correlation Analysis step settings in the lower half of the work area. Click the Minimize preview button (a minus sign icon) at the top right corner of this section.  Alternatively, you can click on More options > Open in a browser tab > Results to open the results in a new browser tab.

Note: If you see a warning message in the results window that says The maximum size limit has been exceeded, you might want to click the Open in new tab link to view the results. Alternatively, go to the SAS Studio Options menu, and select Preferences. Under SAS Programs, click Results. Under Display warning if results are larger than specific size, change the Maximum result size (MB) from 3 to 10.

Each cell in the Correlation table lists several pieces of information: the variable correlated with the response Donation_Amt, the estimate of the correlation coefficient, the p-value, and the sample size. The p-value is for a test of the null hypothesis that the population correlation equals zero. The sample size depends on the number of nonmissing values for only the two variables being correlated. Because of this, the sample size can differ for each correlation.

The correlations have been ranked from strongest to weakest. The strongest correlation with the target is approximately 0.65 for GiftAvgCard36. This positive correlation indicates that as GiftAvgCard36 increases, Donation_Amt tends to increase. It will likely be a good predictor in a regression model for Donation_Amt. The weakest correlation with the target is -0.006 for PromCntCard12. This negative correlation is not statistically different from zero at any commonly used alpha level.

Correlations measure the linear associations between variables. Scatter plots can be useful for assessing linear relationships and thus the appropriateness of correlation analysis. For large data sets, scatter plots can be challenging to interpret when many dots fall on top of each other and are indistinguishable. Prediction ellipses can help you visualize the strength and direction of a correlation. The plots show 95% prediction ellipses for 5 of the 20 variable pairs. The 95% prediction ellipses show where 95% of the data points are expected to lie if both variables are normally distributed. The sign of the correlation becomes obvious from the orientation of the ellipse. The narrower the ellipse, the stronger the linear association between the variables. None of the five plots show strong nonlinear relationships, indicating that correlation is a reasonable tool for describing relationships between these variables. Some plots show unusual data points such as the plot showing Gift Amount Last. Unusual data points, often called outliers, are discussed in a later lesson.

Simple linear regression

Next, we will use a regression node in a flow to regress Donation_Amt onto GiftAvgCard36, the most correlated variable identified earlier in this demonstration. This will produce parameter estimates and hypothesis tests that can be used to understand relationship between variables, as well as a regression equation that can be used to predict future donations from lapsing donors.

1. Return to the previous flow containing the Correlation Analysis step. Make sure the Flow tab is selected and active before proceeding further. From the Steps menu on the far left pane, ensure that you are on the SAS Steps tab. Expand the Statistics group, then double click on the Linear Regression step to add it to the existing flow. Connect the PVA_DONORS node to the Linear Regression node.



2. Click on the Linear Regression node. In the lower half of the work area, ensure the Data tab is selected. Locate the Dependent variable field, then click the plus (+) sign and add the variable Donation_Amt. Click OK.


3. Next, find the Explanatory variables field. Use the plus sign to add GiftAvgCard36.


4. Continue working into the lower half of the work area to configure the remaining settings for the analysis. Click on the Model tab. Leave the default option for Model selection method (Full model fitted (no selection)) and leave the Include intercept option selected.


5. Click on the Statistics tab, under Details on Estimates, check the box for Confidence limits for parameter estimates. Use the default value of 95% for Confidence level.


6. Click on the Plots tab. Under Plots to display, choose Selected plots. Check the boxes next to Observed by predicted values plot, Diagnostic plots, and Scatter plot with regression line.


7. Right click on the Linear Regression node and choose Run node.
   
   Note: You can run a subset of the flow by using the following options:
   
   
   • Run node - runs only the currently selected node in the flow.
   
   
   • Run from node - runs the currently selected node and all nodes that occur downstream from the currently selected node.
   
   
   • Run to node - runs all nodes that occur upstream from the currently selected node. The currently selected node is not run.


8. Save the flow to the VST folder as Correlation and Regression.flw.


9. Click the Minimize Preview button (minus sign icon) located at the top-right corner of the lower half of the work area to collapse the section and then view the results.

In the simple linear regression model, Donation_Amt is regressed on GiftAvgCard36. Only 4111 observations are used. We learned in an earlier lesson that Donation_Amt has missing values for approximately half the observations. Any observation with missing values for the predictor or response is dropped from the analysis.

The Analysis of Variance table shows that the model is significant with P < 0.0001. The R-square indicates that about 42% of the variability in Donation_Amt can be explained by GiftAvgCard36. The adjusted R-square equals 0.42, and the mean square error (MSE) is approximately 392. MSE, also called the error variance, is a measure of the variance in the response variable that is not explained by this model. Both the adjusted R-square and MSE can be used for model comparisons, with higher adjusted R-square or lower MSE indicating a better model.

The Parameter Estimates table shows that the estimated intercept equals 75.41694 and the estimated slope for GiftAvgCard36 equals 1.79672. Thus, the regression model for estimated donation amount is

The p-value for the GiftAvgCard36 parameter indicates that the population slope is significantly different from zero. Thus, this predictor can be useful to predict future donation amounts. The slope of 1.80 indicates that for every additional dollar of GiftAvgCard36, the predicted donation amount will increase by $1.80. The 95% confidence interval on the slope for GiftAvgCard36 indicates the parameter estimate is likely between (1.73, 1.86). The intercept is also significantly different than zero, but tests of the intercept are often less relevant than tests of the slope. In this case, the intercept tells us that the average donation equals $75.42 when GiftAvgCard36 equals zero.

The plot of Observed by Predicted values of Donation_Amt shows that predictions match observed donations poorly when observed donations are under $100.

Several diagnostic plots based on residuals (the difference between observed and predicted values of the response) are shown. These can be used to assess whether the model assumptions are met and to detect extreme observations. Both of these topics are discussed later in this lesson.

The Fit plot shows Donation_Amt plotted against GiftAvgCard36. The best-fit regression line (the solid black line) indicates the predicted value of Donation_Amt for specific values of GiftAvgCard36. The thin light gray shaded region around the regression line indicates the 95% confidence interval on the mean response for specific values of the predictor. It is difficult to see here in the picture Fit Plot, but the confidence intervals are narrowest near the mean values of X and Y and widest at the extremes. The dashed lines show the 95% prediction limits. You can be 95% confident that new observations will lie within these prediction limits.

Talking about multiple regression models with interactions requires some new terminology. Here's a regression model for the PVA_Donors data set.

Donation_Amt = β₀ + β₁ GiftCntCardAll + β₃ GiftCntCardAll * GiftAvgLast + ε

When describing the effect of single predictors on the response, we can refer to these as main effects (for example, the effect of the variables GiftCntCardAll or GiftAvgLast averaged across all other predictors in the model). β₁ and β₂ measure the main effects of the predictors.

Interaction effects refer to the effect of the simultaneous change in two or more predictors. For example, the interaction is modeled as the effect of the product of the two predictors GiftCntCardAll and GiftAvgLast. β₃ measures the interaction effect.

When modeling interactions, you should include the corresponding main effects regardless of whether they are significant. Including the lower order effects is referred to as maintaining model hierarchy, and doing so is beneficial when the goal is statistical inference.

The contour plot below helps visualize the nature of the interaction. The contour lines show the predicted donation amount for specific combinations of the two predictors. How can we interpret this graph?

First, let's get oriented to the graph. The barely visible lowest contour line in the blue-shaded region shows a predicted donation amount of $0. Moving up and to the right on the graph, the lines show $200, $400, $600 and on up to a $1200 contour in the red-shaded region. Without an interaction, the contour lines would be parallel.

Let's look at the left side of the graph and see the effect of GiftAvgLast on Donation_Amt when GiftCntCardAll equals 0. We can see that when GiftAvgLast increases from $0 to $400, the response Donation_Amt also increases from $0 to about $400.

Now, let's look at the right side of the graph and see the effect of GiftAvgLast on Donation_Amt when GiftCntCardAll equals 40. We can see that when GiftAvgLast increases from $0 to $400, Donation_Amt increases from $0 to over $1000.

So clearly, the effect of GiftAvgLast depends on GiftCntCardAll.

Akaike's information criterion (AIC), corrected Akaike's information criterion (AICC), and the Schwarz's Bayesian criterion (SBC, and sometimes called Bayesian information criterion, or BIC) are three of the most used information criteria.

Each information criteria starts with a measure of unexplained variability or error and then adds a penalty component that represents the complexity of the model. The unexplained variability is calculated as n (the sample size) times the natural log of the sum of squared errors divided by n. The magnitude of the penalty is what differentiates each of these information criterion. For model comparison, the smallest information criterion indicates a better model.

These information criteria often suggest the same best model out of a set of candidate models. But when they disagree, SBC tends to favor models with fewer parameters than AIC because of the larger penalty function.

In addition, various criteria can be used to choose the final model out of the sequence of models. The last model in the sequence is not always presented as the "winner." For example, variables could enter or leave the model based on significance level, and then the winning model from the sequence could be chosen based on having the lowest value of an information criterion. There is a lot of flexibility to guide the model selection process.

Problems exist with using stepwise selection methods based on significance level. Some of the problems include p-values that are biased too low, confidence limits for parameter estimates that are biased too narrow, and parameter estimates that are biased away from zero.

What causes these problems? A p-value is intended for evaluating one hypothesis. When it's used for evaluating dozens of models with overlapping sets of predictors, the meaning of the p-value can become distorted.

With these problems in mind, what approaches can be used to find good explanatory models for our data? There's no universally accepted best approach. Prespecifying models based on subject matter expertise instead of using automated model selection approaches can avoid the problems with p-values. But with many predictors, prespecifying the model can become unfeasible.

Information criteria might be a better approach to finding good explanatory models for your data. Also, there are model-averaging approaches that combine several models into one, weighting them based on the AIC. Despite their issues, stepwise methods are often used due to their easy implementation and computational efficiency. However, some authors suggest that shrinkage methods might be superior to stepwise methods of model selection.

So, what are shrinkage methods? Some model selection methods result in parameter estimates that are shrunk relative to the ordinary least squares estimates. Such methods include least angle regression (LAR), elastic net, and LASSO, which stands for least absolute shrinkage and selection operator.

Why would we want to shrink OLS estimates, which are the best linear unbiased estimates?

OLS parameter estimates are unbiased, but they are not necessarily the best estimates to use for prediction. This is because they often produce higher variance predictions due to the trade-off between bias and variance. Although shrinkage methods can add bias to predictions, they can greatly reduce variance and thus prediction error. Although biased parameter estimates are often not ideal for explanatory modeling, some studies have shown that shrinkage methods such as LAR and LASSO do a better job of selecting important predictors and removing noise variables compared with stepwise methods.

You learn more about shrinkage methods and the bias-variance trade-off later in this course.

Demo: Multiple Regression and Model Selection (Self-Paced)

In this demonstration, you use a SAS program to create a multiple regression model for Donation_Amt using the stepwise model selection method. You use Akaike's information criterion (AIC) to add and remove variables from the model. The program Multiple Regression.sas is added to the previously created flow Correlation and Regression.flw.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.


2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.


3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Correlation and Regression.flw from the VST folder. We will add a program to this flow.


2. Navigate to the VST folder and double click on Multiple Regression.sas. Click on the SAS Program step and view the code. While the majority of the code is beyond the scope of this course, note the highlighted portions. The first highlighted section (param=glm) indicates the GLM coding of categorical variables. This default coding creates one numeric (dummy) variable for each level of a categorical predictor. Effect coding (param=effect) and reference coding (param=ref) are other available options. They create one fewer dummy variable than GLM coding and are used for comparing levels to the mean or to a reference level respectively.
   The second highlighted section (selection=stepwise (select=aic)) shows that the multiple regression model uses stepwise variable selection with variables being selected based on Akaike’s information criterion. Here is the code:
   

   ods graphics / imagemap=on; 
   
   
   proc glmselect data=VST.PVA_DONORS outdesign(addinputvars)=Work.reg_design 
   
   plots=(criterionpanel); 
   
   class DemHomeOwner DemGender DemCluster StatusCatStarAll StatusCat96NK/ param=glm; 
   
   model Donation_Amt=GiftCnt36 GiftCntAll GiftCntCard36 GiftCntCardAll GiftAvgLast GiftAvg36 GiftAvgAll GiftAvgCard36 GiftTimeLast GiftTimeFirst PromCnt12 PromCnt36 PromCntAll PromCntCard12 PromCntCard36 PromCntCardAll DemAge DemMedHomeValue DemPctVeterans DemMedIncome DemHomeOwner DemGender DemCluster StatusCatStarAll StatusCat96NK/ showpvalues selection=stepwise (select=aic); 
   
   run; 
   
    
   proc reg data=Work.reg_design alpha=0.05 plots(only)=(diagnostics residuals dffits 
   
   observedbypredicted); 
   
   ods select ParameterEstimates DiagnosticsPanel ResidualPlot DFFITSPlot 
   
         ObservedByPredicted; 
   
   model Donation_Amt=&_GLSMOD / vif; 
   
   run; 



3. Click the + Code to Flow button and add the program to the previously saved Correlation and regression flow. Navigate to the flow tab of Correlation and regression.flw.


4. Right-click on the Multiple regression node and choose Run node. Save the Correlation and regression flow.


5. Click the Submitted Code and Results tab in the upper half of the work area to view the results.

The multiple regression model used 3101 observations due to complete case analysis. The stepwise-AIC model has 26 potential effects and 88 potential parameters. The number of parameters is higher than the number of effects because of the dummy-coding necessary to account for the levels of the categorical predictors.

The stepwise process started with a model containing only the intercept. GiftAvg36 was added at step 1 because it reduced AIC (that is, it improved model fit) more than any other single variable. For steps 2 through 6, adding variables improved AIC more than removing variables. At step 7, a previously entered variable, PromCntCard12, was removed. Then in steps 8–13, variables were added based on producing the greatest reduction in the model AIC.

Model selection stopped at the optimal value of AIC. Any addition or reduction of a single variable from the model at step 13 increased the AIC and made a worse model by this criterion. The Stop Details table shows potential candidate variables to add or remove, both of which increase the AIC, but by a small amount (approximately 0.4).

Fit criteria plots show on the X axis the 13 models corresponding to each step, and on the Y axis one of four fit statistics. Keep in mind that the models assessed during the stepwise selection process are not all possible models, but only a small subset. Of that subset, only fit statistics for the model present at each step are graphed. These graphs show that the final model had the best values of AIC, AICC, and adjusted R-square of any of the 13 models. The best values are indicated by a star. For the SBC, the best model was the model at step 10, which had three fewer inputs than the final AIC model.

The selected model has 11 predictors and is statistically significant (P < 0.0001), meaning at least one predictor in the model has a slope that is significantly different from zero (assuming alpha = 0.05). The R-square indicates that this model explains 54% of the variability in Donation_Amt. The MSE is 302.96 (compare with MSE=392 in the simple linear regression) and the adjusted R-square equals 0.54 (compare with R-square = 0.42 in the simple linear regression). The multiple regression fits the data better than the simple linear regression model. A note indicates that the p-values have not been adjusted for the model selection process. This refers to the p-values being biased too low when used for automated model selection. (For more information about this, see Flom and Cassell 2007).

The Parameter Estimates table shows the estimates for the intercept and the slopes for each input variable. The two categorical variables, StatusCat96NK and StatusCatStarAll, each have multiple parameter estimates, one for each level of the variable except for the reference level. The reference level has a parameter estimate of zero. The other levels have parameters that describe the difference between that specific level and the reference level. For example, the parameter estimate of 2.4977 for StatusCat96NK A indicates a $2.4977 greater average donation amount for members of the A-group than for members of the S- group.

Notice that several of the effects in the model, such as PromCntCardAll and PromCntCard12 are not significant at the alpha=0.05 level. This is because the model selection approach was based on Akaike's information criterion and not significance level. At each step in the model-building process, a variable was either added or removed that improved the AIC (smaller is better) relative to the previous model. At the final step, no further additions or removals of one variable made the AIC smaller. Using other model selection criteria such as SBC, adjusted R-square, or significance level can result in different final models than the stepwise AIC model (not shown). Some of these other models are discussed later in this lesson. The diagnostics plots for this model are discussed in the next demonstration.

The model produced in the demonstration was based on stepwise selection that used AIC to determine which variables entered or left the model. Many other combinations of selection methods and criteria can be used to generate models.

The demonstration model is shown on the first row. The results of three additional approaches are also shown: stepwise selection based on SBC, backward selection based on adjusted R-square, and stepwise selection based on significance level, with a significance threshold of alpha = 0.01. Each of these approaches produced different models that ranged from 6 inputs (SBC) to 15 inputs (adjusted R-square).

Despite different numbers of predictors, the models have similar mean square errors (MSE), meaning that they have similar amounts of variance in the target that they are unable to explain. Lower MSE indicates a better model.

Not surprisingly, the stepwise AIC model is "best" based on the AIC statistic, and the stepwise SBC model is "best" based on the SBC statistic. The backward selection adjusted R-square model has the best value of MSE, R-square, and adjusted R-square out of these four models. The stepwise selection model based on significance level did not have the best fit statistic, but all of the 8 inputs were significant (unlike the AIC model) at the p<0.01 level.

Different model selection methods often suggest different "best" models. These models can all be considered candidate models, and there are usually relatively few compared to the dozens, hundreds, or thousands (or more!) models that were possible candidates before using these model selection methods to narrow your choices.

Sometimes a single fit statistic will be used to select a final model out of the candidates.  But often nonstatistical needs figure into the choice of final model. Now is time to use the goals of your business or research, and your understanding of the field, to make an informed choice when selecting an appropriate model.

What type of criteria might be used to choose a final model in addition to model fit statistics? That depends on how the model will be used.

If data needs to be collected in the future, and the data is expensive or time consuming to collect, a model with fewer inputs might be desirable. In other situations, one model might be chosen over another because the inputs are more reliable. For example, income is easier to measure than job satisfaction. A model with more reliable inputs might be desirable over a model with less reliable inputs, all else being equal. Other times, it is important to include specific predictors. This might be due to wanting to control for those inputs, wanting certain inputs to compare with previous research, or requirements by regulators. The consideration of business needs when choosing a model does not only apply to regression and explanatory modeling. The same considerations show up in using machine learning models for prediction. 

Besides these considerations, the chosen regression model should meet the assumptions of linear regression. These are often assessed through statistical graphics. We talk about the assumptions and the graphs used to assess them in the next section.

How can you reduce collinearity once it's detected?

If you can identify redundancy among independent variables in the model, you might be able to remove one or more of the highly correlated predictors to lessen the multicollinearity. This could involve dropping the variable with the highest VIF until collinearity reaches an acceptable level.

You can also redefine some of the variables as related quantities to reduce their correlation.

Another approach is to use biased regression techniques such as incomplete principal component regression and ridge regression. Linear regression using least squares produces parameter estimates that are unbiased. But in some situations, biased estimates from principal component regression or ridge regression can produce a smaller prediction variance and still provide useful parameter estimates. Because collinearity increases the prediction variance, biased regression techniques can reduce the effect of collinearity. 

Do these approaches apply to analyzing big data or only to explanatory modeling? Different approaches for detecting and removing collinearity are used in big data analysis. One such approach is to remove correlated predictors through variable clustering. A clustering algorithm is applied to the variables before modeling. The algorithm creates clusters of variables that are highly correlated within a cluster but have low correlations between clusters. Then one variable is chosen from each cluster and the rest are removed.

Other common big data approaches are to use partial least squares (PLS) and singular value decomposition (SVD). Partial least squares is a predictive modeling technique where the emphasis is on predictive accuracy and not on understanding the relationships between the predictor variables and the response. Singular value decomposition is an approach related to principal component analysis but is more efficient for large numbers of predictors. PLS and SVD are discussed at length in the Advanced Machine Learning Using SAS Viya course.

Demo: Model Diagnostics (Self-Paced)

In this demonstration, you assess the assumptions for the previously fit stepwise selection model. In addition, you check for collinearity problems and for outliers and influential observations.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Return to the results of the Multiple_regression node in Correlation and regression.flw (last demonstration). Rerun the flow if necessary.
2. View the results on the Submitted Code and Results tab, focusing on the diagnostic tools and plots.




The second Parameter Estimates table has a column for variance inflation factors (VIF). These represent the strength of collinearity for each effect in the model. Collinearity is not a violation of the model assumptions, but it can result in model instability as well as other problems. A VIF >10 indicates a strong collinearity problem with the model.

PromCntCardAll has the highest VIF of 12.46, and GiftTimeFirst has the second highest VIF of 9.48. Collinearity inflates the variance of parameters relative to a model without collinearity, but for both of these variables, collinearity did not inflate the variances enough to make them nonsignificant. Both variables have p-values <0.0001. One approach to addressing collinearity problems is to remove variables involved in the collinearity, usually starting with the one with the highest VIF.





Predicted and observed values of the target match better than in the previous simple linear regression model. In the multiple regression model, smaller donations under $50 are not predicted well. They tend to be overestimated by the model with predictions generally over $50. In addition, higher predicted values tend to be overestimates of the observed donation amount.







The first column of the Fit Diagnostics plots can be used to assess whether the model assumptions have been met. The assumptions of independence, constant variance, and normality are necessary to be able to trust the p-values, standard errors, and confidence intervals from the model.

The residual by predicted value graph shows a non-random pattern. Predicted values over $175 all correspond to negative residuals. This suggests that the model form might be inadequate. Perhaps higher-order terms such as quadratic or interaction effects could improve the model fit. Residual variability might be getting smaller as the predicted values get larger. In addition, the residuals versus normal quantiles and the histogram of the residuals suggest deviations from a normal distribution. The histogram shows left skewness instead of a symmetric distribution. If statistical inference were the goal for this model, the researcher would likely modify this model to better meet the assumptions of regression.

The Cook's D and DFFITS plots show the presence of at least 1 to 3 influential observations. The three spikes that stand out in the Cook's D plot indicate cases that have a large influence on the entire set of parameter estimates (the intercept and the 15 slopes). The same three cases are also influential according to the DFFITS statistic. Large values of DFFITS that are well separated from other such values indicate cases that have a large influence on the predicted values.






The plots of residuals against each of the inputs tend to show the same pattern as residuals versus predicted values: many negative residuals at higher values of predictors and potentially nonconstant residual variance. These plots might indicate a misspecified model. Transformations of the inputs or adding quadratic terms can reduce the pattern of negative residuals at higher values of the predictors.







The last several scatter plots show two columns of dots. These correspond to dummy variables created for the two categorical predictors in the model.

Further effort could be used to improve the fit of this model, or other new models could be created and compared with the stepwise AIC model. These potential next steps are discussed later in this lesson.

What have you learned?

• Correlation describes the linear association between two continuous variables. They are unitless and take values between -1 and +1.
  
  
• Check scatter plots to see whether correlation analysis is appropriate.
  
  
• Correlation alone is not sufficient to infer causality.
  
  
• Linear regression models a response variable, Y, as a linear function of predictors, the Xs.
  
  
• Regression intercept β₀ describes the average Y when X=0.
  
  
• Slope β₁ describes the average change in Y for a unit increase in X.
  
  
• R-square describes the proportion of variability in Y that can be explained by the regression model.
  
  
• Categorical predictors can be used in regression models through dummy coding.
  
  
• Stepwise model selection methods based on p-values or information criteria are computationally efficient methods of searching through vast numbers of models.
  
  
• Regression diagnostics can involve checking assumptions (e ~ i.i.d. N (0, σ²)) and checking for collinearity and influential observations.

The main differences among the analytical methods for predictive modeling depend on the type of target variable. In supervised classification, the target is a class label (categorical). The training data consists of labeled cases. The aim is to construct a model (classifier) that can allocate cases to the classes using only the values of the inputs.

Regression analysis is supervised prediction where the target is a continuous variable. (The term "regression" can also be used more generally; for example, logistic regression is a method used for supervised classification.) The aim is to construct a model that can predict the values of the target from the inputs.

In survival analysis, the target is the time until some event occurs. The outcome for some cases is censored; all that is known is that the event has not yet occurred. Special methods are usually needed to handle censoring.

When making a prediction, a model uses data from the past to make predictions about the future. Time can always be divided into three periods: the past, present, and future.

All three of these periods should be represented in the model set while building the model. Of course, all data comes from the past, so the time periods in the model set are the distant past, the not-so-distant past, and the recent past. Predictive models are built by finding patterns in the distant past that explain outcomes in the recent past. When the model is deployed, it is then able to use data from the recent past to make predictions about the future. Remember the assumption that "future is like the past."

It might not be immediately obvious why some recent data from the not-so-distant past is not used in a particular customer signature. The answer is that when the model is applied in the present, no data from the present is available as input. Any one customer signature should have a gap in time between the predictor variables and the target variable.

This gap is important. For example, if a model were built using data from May (the not-so-distant past) in order to predict June (the recent past), then it could not be used to predict September until August data was available. But when is August data available? Certainly not in August, because it is still being created. Chances are that it won't be available in the first week of September either, because it must be collected, cleaned, loaded, and tested. In many businesses, the August data will not be available until mid-September or even October, at which point nobody will care about predictions for September. The solution is to include a month of latency in the model set. This latency could be thin in case of streaming data and real-time or near-real-time prediction models.

Another aspect is including multiple time frames. The primary goal of the methodology is creating stable models. Even if the model is to be based on only four months of history, different rows of the model set should use different four-month windows. The idea is to let the model generalize from the past rather than memorize what happened at one particular time in the past.

Formally, the expected prediction error is decomposed as shown in the equation here:

E[Prediction Error] = Bias² + Variance + ${\sigma }_e^2$

Note that ${\sigma }_e^2$ is the irreducible error. This is the noise term that cannot fundamentally be accounted for by any model.

The bias is formally the difference between the expected prediction value and the correct response value.

Bias = (Y - E[Predicted Value])

The variance is simply the variability of the prediction values when we construct a model multiple times with different training sets for the same problem, that is, sampling variability. Note that this value of the variance does not depend on the correct response value.

Variance = E[(Predicted Value - E[Predicted Value])²]

Thus, the prediction error for any machine learning algorithm can be broken down into three parts: bias error, variance error, and irreducible error.

Cramer's V statistic is one measure of the strength of an association between two categorical variables, and its value is derived from the chi-square statistic. 

For two-by-two tables, Cramer's V is in the range of -1 to 1, and for larger tables, it's in the range of 0 to 1.

Values farther away from 0 indicate a relatively strong association between the variables. 

The closer Cramer's V is to 0, the weaker the association is between the two variables.

Like other measures of strength of association, the Cramer's V statistic is not affected by sample size.

Demo: Examining Categorical Association (Self-Paced)

In this demonstration, you use the Table Analysis node in a flow to test an association between the two categorical variables, Response and StatusCatStarAll. You also calculate measures of association such as Cramer's V statistic and the odds ratio to determine the strength of association.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.


2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.


3. Click the Run button on the flow toolbar.

Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.


2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.


3. In the work area, click on the Table step. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK.


4. Under SAS Steps, expand Statistics, then double click on the Table Analysis step.


5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Table Analysis node (the input port). Now Table Analysis is connected to the PVA_DONORS data. Click the Arrange nodes button to visually organize the nodes within the flow.


6. Click on the Table Analysis node, then select the Data tab.


7. Under Row variables, click the plus sign and select the StatusCatStarAll variable. Then click OK to close the Column Selection window.


8. Again, click the plus sign next to Column variables and select the Response variable. Then click OK to close the Column Selection window.


9. On the Cell Statistics tab, under Frequencies, select Expected in addition to Observed.


10. Under Percentages, check the box for Row.


11. Under the Association Statistics tab, under Tests of Association, ensure that Chi-square statistics (including Pearson, likelihood ratio and Mantel-Haenszel…) is selected


12. Scroll down to Measures of Association and select Odds ratio and relative risk (for 2x2 tables).


13. On the Computation Options tab, select the option to Suppress plots.


14. Run the flow and save it to the VST folder as Categorical data analysis.flw.


15. Examine the results. The Results window displays the frequency table, which includes the cell frequency, expected frequency, and row percentage.

Examine the row frequency percentages for combinations of values of the two variables. You notice that the row percentages are different in each column, indicating a possible association between the two variables (Status Category Star All Months and Response). To assess whether the differences between the percentages of response to the campaign across the levels of the status category is greater than would be expected by chance, you need to evaluate the chi-square test results.

Because the p-value for the chi-square statistic is <.0001, which is below 0.05, you reject the null hypothesis at the 5% level and conclude that there is evidence of an association between StatusCatStarAll and Response. The Cramer's V statistic value of 0.1106 indicates that the association detected by the chi-square test is relatively weak. The Odds Ratio and Relative Risks table can also be used to measure the strength of association.

The odds ratio is shown in the first row of the table, along with the 95% confidence limits. To interpret the odds ratio, refer to the contingency table at the beginning of the output. The top row (StatusCatStarAll=0, in this case) is the numerator of the ratio, and the bottom row (StatusCatStarAll=1) is the denominator. The interpretation is stated in relation to the left column of the contingency table (Response=No). The value of 1.56 indicates that an individual with a StatusCatStarAll=0 value has about 1.56 times the odds of not responding to the campaign compared to an individual with StatusCatStarAll=1. This is equivalent to saying that an individual with StatusCatStarAll=1 has 1.56 times the odds of responding to the campaign compared with an individual with StatusCatStarAll=0.

Relative risk estimates for each column are interpreted as probability ratios rather than odds ratios. You get a choice of assessing probabilities of the left column (Column 1) or the right column (Column 2). For example, the Column 1 relative risk shows the ratio of the probabilities of StatusCatStarAll=0 to StatusCatStarAll=1 being in the left column (56 / 44.9 = 1.2472).

Demo: Fitting a Multiple Logistic Regression Model (Self-Paced)

In this demonstration, you fit a multiple logistic regression model using the Logistic Regression.sas program in a flow. The program uses Response as the target variable and GiftCnt36, DemMedHomeValue, GiftTimeLast, and StatusCatStarAll as the predictor variables. The goal is to predict whether a customer is likely to donate to the donation campaign.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Categorical Data Analysis.flw from the VST folder. We will add a program to this flow.
2. Still in the VST folder, double click on Logistic Regression.sas. View the code.
3. The SAS Program step contains the following code. Note the oddsratio statement to calculate odds ratios comparing groups with a one-unit difference. Two other statements start with an asterisk, which prevents the code from being executed:






 proc logselect data=VST.PVA_DONORS association;
  partition fraction(validate=0.5 seed=12345);
  class StatusCatStarAll;
  model Response(event='Yes')=StatusCatStarAll GiftCnt36 GiftTimeLast
  DemMedHomeValue / link=logit;
  code file="/home/student/workshop/VST/logistic_score.sas";
  oddsratio;
  *ods select oddsratios;
  *oddsratio StatusCatStarAll GiftCnt36 GiftTimeLast DemMedHomeValue/
    unit (DemMedHomeValue=1000);
run;



4. Click the + Code to Flow button and add the program to the previously saved Categorical Data Analysis flow. Navigate to the flow tab of Categorical Data Anlysis.flw.
5. Right-click on the Logistic Regression node and select Run node. Save the Categorical Data Analysis flow.
6. Examine the results from the Logistic Regression node. If you want to view the results on a separate browser tab, click the More options (three dots) menu and select Open in a browser tab > Results.

On the Results tab, the Model Information table displays information about the model, including the data source, distribution, link function, and optimization technique.




The Number of Observations table contains the number of observations that were read from the source data and used in the analysis. Because the data were partitioned, the table displays the number of observations in each partition.




The Response Profile table shows the distribution of the target variable values for each partition. A Class Level Information table lists the levels of categorical predictors.




The Global Null Hypothesis Test table provides a likelihood ratio test for the hypothesis of whether the final model provides a better fit than a model without effects (an intercept-only model). The p-value (<0.0001) column in the global null hypothesis test table indicates that this model is statistically significant at the 5% level of significance, and at least one of the predictors in the model is useful in predicting Response..




The Fit Statistics table displays a variety of likelihood-based measures of fit. These statistics are useful for assessing the fit of the model to your data. The statistics are computed for each data role when you partition the data.




The Parameter Estimates table shows the parameter estimates, their standard errors, and the p-values for assessing statistical significance. All parameter estimates are on the logit scale. You can use these parameter estimates to define your logistic regression equation.

Because reference cell coding was used for the classification variable, the effect is measured against the reference level. For the StatusCatStarAll variable, level 1 is used as the reference level. So, the estimate for StatusCatStarAll | 0 shows the difference in logits between individuals with 0 and 1 for StatusCatStarAll. The table shows that the parameter estimates for all the effects are statistically significant at the 5% level of significance.




The Association of Predicted Probabilities and Observed Responses table shows that the concordance index (c) equals 0.5938 for the training data and 0.6104 for the validation data. The c statistic value for this model indicates that approximately 60% of the positive and negative response pairs are correctly sorted using StatusCatStarAll, GiftCnt36, GiftTimeLast, and DemMedHomeValue.




The Odds Ratio Estimates table can be used to establish the relationship between predictors and the odds of the event. It is often easier to report odds ratios by first transforming the decimal value to a percent difference value. In this example, the odds ratio estimate for StatusCatStarAll when transformed to percent difference is (0.820-1)*100= -17.96%. In other words, the odds ratio estimate for StatusCatStarAll shows that, when adjusting for other predictor variables, individuals with zero for StatusCatStarAll had ~18% lower odds of donation than the individuals with 1 for StatusCatStarAll. For GiftCnt36, the odds ratio estimate equals 1.084. This means that for each additional donation in the past 36 months, the odds of donation during the campaign changed by a factor of 1.084, an 8.4% increase.

Note: The unusual value of 1.000 for the DemMedHomeValue odds ratio has a simple explanation. Unit (that is, single dollar) changes in the home value do not change the odds of response by an amount captured in three significant digits. To obtain a more meaningful value for this input's effect on response odds, you can calculate an odds ratio comparing median home values that differ by $1000. We’ll alter the code to do this in the next step.

7. Go back to Categorical Data Analysis flow and click on the Logistic Regression node. You will edit the code to change how the oddsratio for DemMedHomeValue is calculated. Add an asterisk to the first oddsratio statement and remove the 2 asterisks from the ods select statement and from second oddsratio statement (which is followed by 4 variable names). Now the oddsratio statement will calculate an odds ratio for DemMedHomeValue, comparing groups with a $1000 difference. The code should appear as follows:






proc logselect data=VST.PVA_DONORS association;
  partition fraction(validate=0.5 seed=12345);
  class StatusCatStarAll;
  model Response(event='Yes')=StatusCatStarAll GiftCnt36 GiftTimeLast
  DemMedHomeValue / link=logit;
  code file="/home/student/workshop/VST/logistic_score.sas";
  *oddsratio;
  ods select oddsratios;
  oddsratio StatusCatStarAll GiftCnt36 GiftTimeLast DemMedHomeValue/
    unit (DemMedHomeValue=1000);
   run;



8. Run the Logistic Regression node again and view the results.



Now the odds ratio for DemMedHomeValue shows a value of 1.001 with a 95% confidence interval of (1.001, 1.002). This indicates that for every $1000 increase in DemMedHomeValue, the odds of donating increase by 0.12%.

Demo: Scoring a Logistic Regression Model (Self-Paced)

In this demonstration, you score a new data set using the score code from the logistic regression model that you created earlier. This enables you to add predictions from a trained model to a score data set.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Categorical Data Analysis.flw from the VST folder. We will add a program to this flow.
Still in the VST folder, double click on Scoring Logistic.sas and view the code.
2. The SAS Program contains the following code. It uses the score code from the logistic regression model to score the SCORE_PVA data set.
3. The predictions are saved to the PVAScored data set:






data VST.PVAScored;
  set VST.SCORE_PVA;
  %include "/home/student/workshop/VST/logistic_score.sas";
run;

proc contents data=VST.PVAScored;
run;



4. Click the + Code to Flow button and add the program to the previously saved Categorical Data Analysis flow. Navigate to the flow tab of Categorical Data Anlysis.flw.
5. Right-click on the Scoring Logistic node and select Run node. Save the Categorical Data Analysis flow.
6. Examine the Results as well as the Output Data tab. Note the variables P_response and I_response in both the Results and the Output Data tabs

Artial view of the output table (PVASCORED) is shown below:

The P_Response column shows the predicted probability of the target Response=Yes. The I_Response column is an indicator variable that takes the value of Yes when the value of P_Response is greater than or equal to 0.5 and No otherwise.

Note: Your output might show a different listing of scored data.

What have you learned?

• The goal of explanatory modeling is causal explanation. The goal of predictive modeling is empirical prediction.
  
  
• After a model and allocation rule are determined (training), the model must be applied to new cases (scoring).
  
  
• For scoring, SAS Viya translates the model into a format called score code, which is a SAS program.
  
  
• Scoring must incorporate all data manipulation tasks done before generating the model.
  
  
• Predictive modeling typically involves choices from among a set of models. These might be different types of models, or they might be different complexities of models of the same type.
  
  
• Selecting model complexity involves a trade-off between bias and variance. Both bias and variance are forms of prediction error in machine learning.
  
  
• An overly simple model might not be flexible enough and generally leads to underfitting. An overly complex model might be too flexible and generally leads to overfitting.
  
  
• An optimal balance of bias and variance would never overfit or underfit the model.
  
  
• An association exists between two categorical variables if the distribution of one variable changes when the value of the other variable changes.
  
  
• Chi-square test is commonly used for feature selection in machine learning.
  
  
• You use the odds ratio to measure the strength of the association between a binary predictor variable and a binary response variable.
  
  
• A logistic regression model applies a logit transformation, or simply the log odds transformation, to the probabilities.
  
  
• The c value is one of the most used measures of model performance.
  
  
• You need to determine the best cutoff in logistic discrimination.

Reinforcement learning has three primary components: the agent (the learner or decision maker), the environment (everything the agent interacts with), and actions (what the agent can do). The objective is for the agent to choose actions that maximize the expected reward over a given amount of time. The agent will reach the goal much faster by following a good policy. So, the goal in reinforcement learning is to learn the best policy.

Reinforcement learning can probably be best explained through an analogy to a video game. As a player progresses through the virtual space of a game, they learn the value of various actions under different conditions and become more familiar with the field of play. Those learned values then inform and influence a player's subsequent behavior, and their performance immediately improves based on their learning and experience.

Apart from neural networks, some other machine learning algorithms are categorized into four common groups: pattern discovery, feature extraction, estimation and predictive modeling. They are described below briefly.

Association rule discovery, or, market basket analysis, is used to analyze streams of transaction data (for example, market baskets) for combinations of items that occur (or do not occur) more (or less) commonly than expected. Retailers can use this to identify interesting combinations of purchases or as predictors of customer segments. Sequence analysis is an extension of market basket analysis to include a time dimension to the analysis. In this way, transactions data is examined for sequences of items that occur (or do not occur) more (or less) commonly than expected. A webmaster might use sequence analysis to identify patterns or problems of navigation through a website.

The k-Means Clustering algorithm is a type of unsupervised learning, which is used to categorize unlabeled data, that is, data without defined categories or groups. The algorithm works by finding groups within the data, with the number of groups represented by the variable k. It then works iteratively to assign each data point to one of k groups based on the features provided.

Self-organizing maps are unsupervised data visualization techniques that reduce dimensionality of data using self-organizing neural networks (Kohonen Network). The networks are trained to reduce dimensions of data by mapping high-dimensional data on a low-dimensional grid of neurons. They display and show the similarities among observations in the reduced dimensions by using a rectangular (eight neighbors) or hexagonal grid (six neighbors) of neurons in the output layer.

The expectation-maximization (EM) algorithm (Dempster, Laird, and Rubin 1977) is a tool that addresses problems of missing data. The EM algorithm proceeds by finding the conditional expectation of the missing data, given the observed data, treating that conditional expectation as observed data, and iterating until the model converges.

The k-Nearest-Neighbor algorithm estimates how likely a data point is to being a member of one group or another. It essentially looks at the data points around a single data point to determine what group it is actually in. For example, if one point is on a grid and the algorithm is trying to determine what group that data point is in (group A or group B, for example) it would look at the data points near it to see what group most of the points are in.

Principal Component Analysis (PCA) attempts to find a series of orthogonal vectors that better describe the directions of variation in the data than the original inputs do. A geometric interpretation of orthogonal is that the vectors are perpendicular; a statistical interpretation is that the vectors are uncorrelated. The goal is to be able to characterize most of the variation in the data with as few vectors as possible.

Singular Value Decomposition (SVD) decomposes a matrix into special matrices that are easy to manipulate and analyze. It projects high-dimensional data into a lower-dimension space that results into independent linear components. PCA skips less-significant components. You can use SVD to find PCA by truncating the less-important basis vectors in the original SVD matrix.

Variable clustering is a useful technique for data reduction because it finds the best variables for analysis. It removes collinearity, decreases variable redundancy, and helps reveal the underlying structure of the input variables in a data set in the sense that the groups of variables reveal the main dimensionalities of the data. 

In partial least squares regression (PLS) modeling, model weights are chosen to simultaneously account for variability in both the target and the inputs. It generally produces meaningful predictions based on limited data (for example, when there are more inputs than modeling cases). A variable selection method named variable importance in the projection arises naturally from the PLS framework.

An autoencoder is a special form of neural network that is trained not to try to map input values to output values, but instead to map input values to themselves. Autoencoders are used to create abstractions called encoders that are created from a given set of inputs.

Local search is a heuristic method for solving computationally hard optimization problems. Local search can be used on problems that can be formulated as finding a solution maximizing a criterion among several candidate solutions. Local search optimization begins concurrently for each selected point.

Kernel density estimation is a nonparametric technique for density estimation in which a known density function (the kernel) is averaged across the observed data points to create a smooth approximation.

Linear regression is the most basic type of regression. Simple linear regression allows us to understand the relationships between two continuous variables.

Logistic regression focuses on estimating the probability of an event occurring based on the previous data provided. It is used to cover a binary dependent variable, that is where only two values, 0 and 1, represent outcomes.

A decision tree is a flow-chart-like tree structure that uses a branching method to illustrate every possible outcome of a decision. Each node within the tree represents a test on a specific variable – and each branch is the outcome of that test.

Forests or random decision forests is an ensemble learning method, combining multiple algorithms to generate better results for classification, regression and other tasks. Each individual classifier is weak, but when combined with others, can produce excellent results. The algorithm starts with a decision tree (a tree-like graph or model of decisions) and an input is entered at the top. It then travels down the tree, with data being segmented into smaller and smaller sets, based on specific variables.

Adaptive regression splines method is a nonparametric regression technique that combines both regression splines and model selection methods. It does not assume parametric model forms and does not require specification of knot values for constructing regression spline terms. Instead, it constructs spline basis functions in an adaptive way by automatically selecting appropriate knot values for different variables and obtains reduced models by applying model selection techniques. 

A Bayesian network is a directed acyclic graphical model that represents probability relationships and conditional independence structure between random variables. For example, the Naïve Bayes classifier is based on Bayes' theorem and classifies every value as independent of any other value. It allows us to predict a class/category, based on a given set of features, using probability. Despite its simplicity, the classifier does surprisingly well and is often used due to the fact it outperforms more sophisticated classification methods.

The Gaussian mixture model is a probabilistic model that assumes that all the data points are generated from a mixture of Gaussian distributions; see Rasmussen (1999). In probabilistic modeling, the probability distribution over all the discovered clusters is inferred for each observation. Thus the Gaussian mixture model provides soft clustering, which is different from the hard clustering that is provided by k-means, hierarchical, and density-based spatial clustering.

Rule Induction combines decision tree and neural network models to predict nominal targets. It is intended to be used when one of the nominal target levels is rare. New cases are predicted using a combination of prediction rules (from decision trees) and a prediction formula (from a neural network, by default). Input selection and complexity optimization are described below.

An artificial neural network (ANN) is essentially many interconnected processing elements, working in unison to solve specific problems. ANNs are inspired by biological systems, such as the brain, and how they process information. ANNs also learn by example and through experience, and they are extremely useful for modelling non-linear relationships in high-dimensional data or where the relationship among the input variables is difficult to understand.

Gradient boosting is an iterative approach that creates multiple models, preferably decision trees, on the training data. The gradient boosting model improves its predictions by minimizing a specified loss function, such as average square error. The first step creates a baseline tree. Each subsequent tree is fit to the residuals of the previous tree, and the loss function is minimized. This process is repeated a specific number of times. The final model is a single function, which is an aggregation of the series of trees that can be used to predict the target value of a new observation.

Support Vector Machine algorithms are supervised learning models that analyze data used for classification and regression analysis. They essentially filter data into categories, which is achieved by providing a set of training examples, each set marked as belonging to one or the other of the two categories. The algorithm then works to build a model that assigns new values to one category or the other.

Memory Based Reasoning (MBR) is the implementation of k-nearest neighbor prediction on nominal target. Decisions are made based on the prevalence of each target level in the nearest k-cases. Most primary outcome cases results in a primary decision, and most secondary outcome cases results in a secondary decision. Complexity of the model can be adjusted by changing the number of neighbors considered for prediction.

Least angle regression was introduced by Efron et al. (2004). Not only does this algorithm provide a selection method in its own right, but with one additional modification it can be used to efficiently produce LASSO solutions. Just like the forward selection method, the LAR algorithm produces a sequence of regression models where one parameter is added at each step, terminating at the full least squares solution when all parameters have entered the model.

The ensemble approach relies on combining many relatively weak simple models to obtain a stronger ensemble prediction. Ensembling creates a new model by combining the predictions from multiple models to improve accuracy, reduce bias, reduce variance, and provide robust models in the presence of new data. The commonly observed advantage of ensemble models is that the combined model is better than the individual models that it consists of.

Let's see how a user can interact with a parallel coordinate plot and extract desired insights. To view the polylines for a single group only, you select that group on the far left. Notice that the polylines for all other groups are dimmed. Users have the flexibility of selecting multiple groups. This enables you to focus on the group of interest and find relationship among variables.

Furthermore, you can adjust the range of a quantitative variable to see how those values are distributed among other variables. In this example, you see the distribution of income and home value variables for a specific age group of individuals (say 70-87 years in this case) across the levels of gender variable. You can repeat this step for multiple variables.

Demo: Modifying and Correcting Data (Self-Paced)

In this demonstration, you use a flow with the t-Tests node to test whether the population mean donation amount is $100 given the information in the PVA_DONORS table.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.
3. In the work area, click on the Table node. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK.
4. Under SAS Steps, expand the Transform Data group, then double click on the Recode Values step. Right-click the Recode Values node and choose Add output port.
5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Recode Values node (the input port). Now Recode Values is connected to the PVA_DONORS data.
6. Add a second Table node to the flow. For the second Table, enter VST for Library and MISSING_PVA for Table name.
7. Now connect the box on the right side of the Recode Values node (the output port) to the MISSING_PVA node. Click the Arrange Nodes button to visually organize the flow.





8. Click on the Recode Values node. Under the Data tab, the Numeric variable radio button is selected by default. Click on the plus sign (+) next to Variable to recode. Choose DemMedIncome and click OK.
9. Click on the Values tab. Under Recode values, enter zero (0) in the Old value field. In the New value field, add a single period (.) which indicates missing values.
10. Click on the Output tab. In the Recoded variable name field, change the default name to Rep_DemMedIncome. Ensure that Write to another data set and Replace existing output table are both selected.

Run the flow. Save the flow in the VST folder as Data Preprocessing.flw.

11. After the flow runs successfully, click the Submitted Code and Results tab. Click the Output Data tab to view the Missing_PVA data set and scroll to the right. You see that a new column, Rep_DemMedIncome, has been created that has missing values against the zero values present in the DemMedIncome column.

Missing values arise for a variety of reasons.

One might be due to no match on merge. In business data, we're often merging tables together. Well, if we've got a customer that exists in one data table, but she does not exist in the other data table, when we merge the data together, that customer is going to have missing information about some of the variables.

Another reason we might encounter missingness is non-disclosed measurement. Some information like age, income, and net wealth, people don't want to self-report that information. So, if we're expecting customers to provide that information for us and they don't, again, we're going to have missing values.

And another reason we might encounter missingness is due to something that doesn't apply. This is referred to as a non-applicable measurement. For example, asking "Have you ever been pregnant?" to a male patient or asking "Is this your first VIP purchase?" to a customer who is not enrolled in the VIP program.

Demo: Managing Missing Values (Self-Paced)

In this demonstration, you learn to address missing values in the MISSING_PVA data set. The data set has quite a few variables with missing values. However, for brevity, let's focus on the Rep_DemMedIncome variable. Use the Imputation node in a flow to impute synthetic data values.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Data Preprocessing.flw from the VST folder.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, expand Prepare Data, then double click on the Imputation step.
3. Right click on the Imputation step and choose Add output port.
4. Next, move your mouse towards the edge of the MISSING_PVA node until the pointer becomes a hand icon. Click and drag an arrow from MISSING_PVA node to the box on the left side of the Imputation node (the input port). Now Imputation is connected to the MISSING_PVA data.
5. Add another Table step to the flow by double clicking Table under Data (Input and Output). For this node, under Table Properties, enter VST for Library and IMPUTED_PVA in the Table name field.
6. Connect the output port of the Imputation node to the IMPUTED_PVA node. Use the Arrange nodes button to visually organize the nodes in the flow.




7. Click on the Imputation node, then select the Data tab.
8. Under Interval Variables, click the plus sign corresponding to Replace missing values with the mean and select the Rep_DemMedIncome variable. Then click OK to close the Column Selection window.

Note: Users also have the option to impute the missing values in interval variables with the median or a random number. If required, users can choose different methods for different variables.

Note: The Imputation step also supports imputing missing values in nominal variables by mode.
9. Click the Output tab. Make sure Save imputed data and Replace existing output table are both selected.

Note: To expedite the demonstration in the class setup, a CAS table in the VST library has already been saved with the same name. Thus, the Replace option is used.

10. Next, to include all the variables from the input CAS table in the output CAS table, click the All variables radio button under Include variables from the input CAS table.
11. Click Run to execute the flow. Save the file Data Preprocessing.flw. When the run is complete, view the results in the work area. You clearly see that a new input, IM_Rep_DemMedIncome, is added to the data set, and it has missing values replaced by a synthetic value ($53513, the mean in this case) and nonmissing values copied from the original input.





12. Click the Output Data tab to view a snapshot of the output table. Scroll your cursor to the right, and you see the new column, IM_Rep_DemMedIncome, that contains the value 53513 for every missing value in the Rep_DemMedIncome column. Because both the versions of the income variables are available in the data set, going forward, you should use the IM_Rep_DemMedIncome variable for analysis.

Demo: Transforming Inputs (Self-Paced)

Transforming inputs can sometimes improve the fit of a model to the data. Recall that in our data set we have several gift-related variables that are heavily skewed. In this demonstration, you learn to transform such variables using a pre-created flow to regularize their distribution. For brevity, we will consider just one of the gift-related variables in this demonstration.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Data Preprocessing.flw from the VST folder.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, expand Prepare Data, then double click on the Transform columns step.
3. Next, move your mouse towards the edge of the IMPUTED_PVA node until the pointer becomes a hand icon. Click and drag an arrow from IMPUTED_PVA node to the box on the left side of the Transform Columns node (the input port). Now Transform Columns is connected to the IMPUTED_PVA data.
4. Add another Table step to the flow by double clicking Table under Data (Input and Output). For this node, under Table Properties, enter VST for Library and PVA_DONORS_FINAL in the Table name field.
5. In the work area, connect the output port of the Transform Columns node to the PVA_FINAL_DONORS node. Click the Arrange nodes button to visually organize the flow.
6. Click on the Transform Columns node. Under the Data tab, Find Select a server for this step and choose the CAS radio button.
7. Scroll down to the Variable 1 field under Transform 1. Click the plus sign (+) and choose GiftCnt36. Click OK to close the Column Selection window.
8. In the Transform field under GiftCnt36, use the drop-down arrow to choose Specify custom transformation. In the Custom transformation field, type in log(GiftCnt36+1).






Note the +1 added to the variable GiftCnt36. Some donors did not donate in the past 36 months, so have a GiftCnt36 value of zero. The log of zero is undefined, so adding 1 to GiftCnt36 prevents the transformation from generating missing values when GiftCnt36=0.

9. Click on the Output tab. Make sure Replace existing output table is selected. Under Specify the data to show in the Results tab, use the dropdown menu to select Specific number of observations. Leave the default number of observations set to 10.
10. Run the flow and save Data Preprocessing.flw. View the Results.

Partial view of PVA_DONORS_FINAL:







The Results tab in the work area shows the first ten rows of PVA_FINAL_DONORS. Note that you may see a different 10 rows on your computer. You can see that a new input named tr1_GiftCnt36 has been added to the table.

Going forward you might want to use this log transformed variable over the original variable.

Demo: Performing Feature Selection (Self-Paced)

Removing redundant and irrelevant inputs from a training data set often improves the performance of a model. In this demonstration, you use a pre-created flow to select potentially useful features (variables).

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Data Preprocessing.flw from the VST folder.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, expand Prepare Data, then double click on the Variable Selection step.
3. Next, move your mouse towards the edge of the PVA_DONORS_FINAL node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS_FINAL node to the box on the left side of the Variable Selection node (the input port). Now Variable Selection is connected to the PVA_DONORS_FINAL data. Click the Arrange nodes button to visually organize the flow.
4. Click on the Variable Selection node. Under the Data tab, ensure that the Unsupervised variable selection radio button has been selected.
5. Expand Explanatory Variables using the dropdown arrow. Under Explanatory Variables, find the field for Continuous variables and click the plus sign (+) to add numeric columns to the variable list. Select all the numeric inputs except GiftCnt36, StatusCatStarAll, DemMedIncome, Donation_Amt, and Rep_DemMedIncome. You can do so by clicking the Select all check box next to the Name field to include all the numeric inputs and then clear the boxes corresponding to GiftCnt36, StatusCatStarAll, DemMedIncome, Donation_Amt, and Rep_DemMedIncome. Click OK to close the Column Selection window.
6. Scroll down to the Classification variables field. Click the plus sign (+) to add variables. In the Column Selection window, select StatusCat96NK, StatusCatStarAll, DemGender, and DemHomeOwner. Click OK to close the Column Selection window.
7. Click on the Model tab to view these settings. Leave the default option Main effects model selected.
8. Next, click on the Options tab. Leave the default settings for Select variables based on: Correlations, Proportion of total variance to be explained: Use 0.9, and Maximum number of selected effects: Use 200.
9. Scroll down and select the box next to the Specify minimum increment of explained variance property. Verify that the Increment of explained variance value is set at 0.001.
10. Finally, under the Plots option, select the Iteration plot box to include an iteration plot in the results.
11. Typically, an output port connected to a table node would be added to Variable Selection to save the variable list to a data set. In the interest of time, here we will just right-click on the Variable Selection node and choose Run node to view the variable list in the Results. Save Data Preprocessing.flw and view the results.

When the run is complete, the results are displayed in the work area. Note that the variable reduction algorithm has used 5908 observations out of a total of 9686 observations. This is because of the presence of several missing values. Remember that, for brevity, you imputed the missing values in just one of the several variables. In practice, you should impute the missing values in all the variables before performing variable selection.

The Selection Summary table lists the variables in the order of selection. The variable that explains most of the variability in the input space is selected first. In the current example, Promotion Count All (PromCntAll) explains around 20% of variability, which is the highest among all inputs. The selection process stops after 17 iterations. Can you guess why? Well, the final parameter added to the selection summary resulted in just over 90% of the original variability being explained, triggering the algorithm to stop.







Next you see the Selected Effects table, which provides a summary of selected variables. These are the 12 variables that you might want to use as features in any supervised machine learning algorithm that follows.







When you scroll down farther in the results window, you see the iteration plot that summarizes the incremental variance explained by adding a variable in each iteration. This is essentially a grouped bar chart of increasing size in which blue bars represent the base value of variance explained and yellow bars represent the incremental value.

In the case of L1, the sum of the absolute values of the weights is imposed as a penalty, and in the case of L2, the sum of the squared values of weights is imposed as a penalty.

In L1 regularization, the weights shrink by a constant amount toward 0. In L2 regularization, the weights shrink by an amount that is proportional to square of weight. And so, when a particular weight has a large magnitude, L1 regularization shrinks the weight much less than L2 regularization does. By contrast, when weight is small, L1 regularization shrinks the weight much more than L2 regularization.

The L1 penalty has the effect of forcing some of the coefficient estimates to be exactly zero when the tuning parameter λ is sufficiently large. Therefore, the lasso method also performs variable selection and is said to yield sparse models. L2 regularization shrinks the coefficients for less important predictors, very close to zero, but not exactly to zero. Hence, it generates a dense model (having all predictors) that makes it less interpretable. On the other hand, L1 penalty has the effect of forcing some of the coefficient estimates to be exactly equal to zero, thereby resulting in a simple and interpretable model. Therefore, the lasso method also performs feature selection.

We've discussed the definitions of model parameters and hyperparameters - and the difference between them. In brief, model parameters are internal to the model and estimated from data automatically. Hyperparameters are set manually, are used in the optimization of the model, and help in estimating the model parameters.

If your predictions are totally off, your loss function will output a higher number. If they're pretty good, it'll output a lower number. As you change pieces of your algorithm to try to improve your model, your loss function will tell you if you're getting anywhere.

The error function is always a deviance function. An error function measures the deviation of an observable value from a prediction, whereas a loss function operates on the error to quantify the negative consequence of an error. Here it might be reasonable to posit that there is squared error loss, where the negative consequence of an error is quantified as being proportional to the square of the error. In other contexts, you might be more negatively affected by an error in a particular direction (for example, false positive versus false negative), and, thus, you might adopt a non-symmetric loss function. The error function is a purely statistical object, whereas the loss function is a decision-theoretic object that you are bringing in to quantify the negative consequences of error. The loss function is a value that is calculated at every instance. The cost function is usually more general. The cost function is calculated as an average of loss functions over your training set. In short, we can say that the loss function is a part of the cost function.

Demo: Running a Neural Network Model and Tuning Its Hyperparameters (Self-Paced)

In this demonstration, you use a supervised machine learning algorithm to train a neural network a model. Using a pre-created flow, you also partition the data to perform honest assessment. Finally, you tune the hyperparameter values to improve the performance of the model.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Navigate to the VST folder and double click on Neural_Network_PVA_Final.flw.
2. Click on the SAS Program step and view the code. While the majority of the code is beyond the scope of this course, note the highlighted portions. The partition statement partitions the data into 50% for training and 50% for validation. The hidden statement creates 50 hidden units in the neural network. The REGL2=0.1 option in the OPTIMIZATION statement is used to apply an L2 (ridge) regularization. This adds a penalty to the sum of the squared slopes, which helps prevent overfitting by discouraging overly complex models. Here is the code:






proc nnet data=VST.PVA_DONORS_FINAL standardize=std; 
	partition fraction (validate=0.5 seed=12345); 
	target Response / level=nominal; 
	input GiftAvg36 PromCntAll PromCnt36 / level=interval; 
	input StatusCat96NK DemGender DemHomeOwner / level=nominal; 
	hidden 50; 
	train stagnation=5 outmodel=VST.Nnet_model; 
	optimization regL2=0.1; 
	*train stagnation=5 outmodel=VST.Nnet_model2; 
	*optimization regL2=0.001; 
run; 




In a practical scenario, you should consider all 12 features selected in the previous demo performing variable selection, but for the purposes of brevity, here we will be using only the top 6 features (inputs).

3. Click Run to execute the flow and open the results in a new browser tab.

The Model Information table includes information about the number of observations used to train the model, the number of hidden layers, hidden nodes, and the number of parameters estimated.



Note that the misclassification rate for validation data is 0.4511. Because the target variable is nominal, the algorithm tries to choose parameter estimates that can optimize the estimation criterion, which is the misclassification rate in this case.

The Iteration History table indicates that the algorithm performed 9 iterations in all, with the final model having the smallest validate error value (0.4511).

Note: Some SAS Visual Data Mining and Machine Learning models (including the neural network) are created using a nondeterministic process. This means that you might experience different results when you rerun the task.



To improve the performance of the model, you might want to add more inputs or tune the hyperparameter values, or both.

Next you will edit some of the hyperparameters.

4. Return to the flow and update the L2 hyperparameter from 0.1 to 0.001. To make this change, place an asterisk (*) before the first OPTIMIZATION statement and remove the asterisk from the second OPTIMIZATION statement. Do the same for the TRAIN statements: place an asterisk before the first statement and remove the asterisk from the second statement. This change will save the updated neural network as Nnet_model2 in the VST library. The code will now appear as follows:






proc nnet data=VST.PVA_DONORS_FINAL standardize=std;
	partition fraction (validate=0.5 seed=12345); target Response / level=nominal;
	input GiftAvg36 PromCntAll PromCnt36 / level=interval;
	input StatusCat96NK DemGender DemHomeOwner / level=nominal;
	hidden 50;
	*train stagnation=5 outmodel=Nnet_model;
	*optimization regL2=0.1;
	train stagnation=5 outmodel=VST.Nnet_model2;
	optimization regL2=0.001;
run;



5. Click Run to execute the flow and view the Results.

The Model information table indicates that the chosen model is the one with a misclassification rate on validation data equal to 0.4317, which is slightly lower than that of the previous model.

The Iteration History table indicates that the algorithm has performed as many as 103 iterations in the search for an optimal solution. However, the optimal solution was obtained at iteration number 53, producing the smallest misclassification rate on the validation data.



Note that the result produced by the neural network is not as interpretable as the regression models you saw earlier. This is often cited as one of the limitations of machine learning models, a topic that is discussed in the next section.

The interpretability of a machine learning model is essential for gaining insight into model behavior. Understanding the rationale behind the model's predictions would certainly help users decide when to trust or not to trust their predictions, which is fundamental if one plans to act based on a prediction, or when choosing whether to deploy a new model. Model interpretability can meet different needs for different users, such as regulators, executives, data scientists, and domain experts.

• Regulators need model interpretability to make sure the model makes predictions for the right reasons. For example, if an individual's loan application is rejected, the loan agency needs to confirm that this decision does not violate any laws that protect certain groups of people.
• Executives need to understand black-box models so that they can logically justify the decisions that they make.
• Lastly, data scientists (with the help of domain experts) need model interpretability to be able to detect biases that exist in training data, extract new knowledge that is originally hidden in the data, and debug models when they produce wrong and unexpected predictions.

All those users want similar things from the black box models. They want them to be transparent, trustworthy, and explainable.

• Transparent: The model can explain how it works or why it gives certain predictions, or both.
• Trustworthy: The model can handle different scenarios in the real world without continuous control.
• Explainable: The model can convey useful information about its inner workings, for the patterns that it learns and for the results that it gives.

What have you learned?

• Supervised, unsupervised, semi-supervised, and reinforcement learning are different approaches of machine learning.
  
  
• A neural network can be thought of as a regression model on a set of derived inputs, called hidden units.
  
  
• Data preparation tasks, like formatting and joining of tables, convert a raw table to an analytical base table.
  
  
• Data preprocessing tasks, like feature engineering, convert an analytical base table to a model development table.
  
  
• Like any other data analysis approach, machine learning poses several data difficulties and modeling challenges.
  
  
• You need effective data visualizations while exploring the data before fitting models as well as after fitting models for model diagnostics.
  
  
• You must think beyond traditional statistical plots for effective visualization of big data.
  
  
• Errors, outliers, and missing values must be detected, investigated, and corrected, if possible.
  
  
• If you do not impute missing values, complete case analysis can significantly reduce the analysis data and can bias the results.
  
  
• Missingness can be handled using several imputation strategies like fixed value imputation, tailored-value imputation, and cluster imputation.
  
  
• Outliers might not be problematic for all machine learning models. Outliers are not always awkward.
  
  
• Binning and suitable transformations can mitigate the influence of outliers or extreme observations in inputs.
  
  
• Redundancy, irrelevancy, curse of dimensionality, sparseness, and overfitting are common problems with too many input variables.
  
  
• Use feature engineering tasks, like feature construction, feature selection, and feature extraction, to determine the best set of features for model building.
  
  
• Ensuring features within a similar scale is imperative. Use feature scaling techniques like standardization, normalization, and mid-range.
  
  
• Some machine learning algorithms, like neural networks, are sensitive to feature scaling. Others, like decision trees, are virtually invariant to it.
  
  
• Use resampling methods, like cross validation and bootstrapping, when you encounter small data sets.
  
  
• Machine learning algorithms often result in overly complex models. However, optimum complexity can be achieved by regularization methods.
  
  
• Almost every machine learning algorithm is an optimization problem that is solved iteratively.
  
  
• SAS Viya can automate the selection of hyperparameters.