Course Code: EVST151, prepared date: November 24, 2025
Copyright ©
SAS Institute Inc., Cary, NC, USA. All rights reserved.

## Lesson 01

## Statistics You Need to Know for Machine Learning
Lesson 01, Section 1 Demo: Automated Machine Learning in SAS Studio

## Demo: Automated Machine Learning in SAS Studio (Self-Paced)

In this demonstration, you access and examine the SAS Studio interface. You then load the course data in memory by using a pre-created SAS Studio flow. By running this flow, you establish a CAS session and then you connect your session to the location of pre-existing data files to load them into memory. Any data processing in SAS Viya requires the data to be loaded in memory first. Finally, you create a flow with the Automated Feature Engineering step to generate features from the PVA_Donors data that can be used as inputs to a predictive model.

Examining the SAS Studio Interface

1. From the Windows taskbar, launch Google Chrome. When the browser opens, select SAS Landing from the bookmarks bar. The user ID and password should be prefilled, but if not, sign in as student using the password Metadata0. (Be sure to use an uppercase M in the password.) If prompted, select Yes for assumable groups.

Note: You can also directly select the SAS Studio page from the bookmarks bar, if available.

2. Click the Applications menu button (the three-by-three grid) in the upper left corner, expand ANALYTICS LIFE CYCLE, and select Develop Code and Flows to launch SAS Studio. The SAS Studio Develop Code and Flows window is displayed. The main window of SAS Studio consists of a navigation pane on the left and a work area on the right. The navigation pane provides easy access to your snippets, steps, and libraries, as well as other components. The work area is used to display your data, code, logs, results, and flows.
3. In the left navigation pane, click the Explorer icon (second from the top). Programs and Flows will be run from the VST folder (Files/Home/workshop/VST). Note: The path to the VST folder may change in later versions of SAS Viya.
4. Next, click the Steps icon (third from the top). Steps can be used to create flows which automate data manipulation and analysis tasks.
5. Now click the Libraries icon (second from bottom) to access the default SAS libraries. You see the WORK, SASHELP, and SASUSER libraries, as well as others. Next, you create and access custom caslibs.

Assigning Libraries and Loading Course Data

Next you run the pre-created SAS Studio flow to assign libraries and load course data. A flow consists of one or more objects. Each object is represented by a node in the process flow. The flow shows the relationship between two or more objects such as a SAS program, a task, a query, and so on. You access pre-created flow as follows:

1. In the left navigation pane, click the Explorer icon (second from top). Expand Files > Home > workshop > VST > AssignLibrary.flw and double-click Assign Library flow.

Note: The SAS Program node in this flow executes a SAS program behind the scenes to establish a CAS session, assign libraries, and load the data in memory. A discussion about the underlying code in the flow is beyond the scope of this course.

2. Run the flow by clicking the Run toolbar button.
3. A green check mark on the Assign Library node indicates that the flow has run successfully. Click the Libraries icon (second from bottom) in the left navigation pane. Look for the VST library and expand it to ensure that the PVA_DONORS table along with other data tables are loaded in memory and are accessible via the VST library. With this step completed, one of the important tasks of loading the data into memory has been accomplished. You analyze this PVA_DONORS data set in upcoming lessons. However, to start with, let's move directly into automated machine learning in SAS Viya.

Note: A pre-made flow with steps for all the demonstrations in this course is located in the VST/backup flows folder.

Optional: After initially running AssignLibrary.flw, this flow is used later to re-establish the VST library whenever SAS Studio times out. As an optional alternative approach, the code from this flow can be added to the Autoexec file. Code in the Autoexec file is executed whenever SAS studio is started. This will result in the VST library being established whenever the user logs into SAS studio. To do this, click on Options at the top left side of SAS Studio, then choose Autoexec file. Copy and paste the code from the AssignLibrary flow to the Autoexec window, then click Save. The VST library will now be available whenever you log into SAS Studio.

Automated Machine Learning

1. In the work area, click the plus sign next to the AssignLibrary.flw tab to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps tab, expand Data (Input and Output) group, double click on the Table step.
3. In the work area, click on the Table step. At the lower half of the work area, under Table Properties tab, click the file cabinet next to the Library field. Under Libraries on the left, click on the VST library, then select the PVA_DONORS data on the right and click OK.
4. Scroll down through SAS Steps and expand Machine Learning group. Double click on theAutomated Feature Engineering step. The Automated Feature Engineering step explores the variables in the input data and automatically performs feature engineering in a parallel, scalable way.
5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Automated Feature Engineering node (the input port). Now Automated Feature Engineering is connected to the PVA_DONORS data. Click the Arrange nodes button to visually organize the nodes within the flow.
6. Click on the Automated Feature Engineering node. in the lower half of the work area, under the Data tab, find the Target field. Make sure Use a nominal target radio button is selected. Use the plus sign to select a nominal target variable. Scroll down the variable list and click on Response, then choose OK.
7. Under Inputs, ensure that Use all columns except target is selected.
8. Click the Options tab. Keep the settings under Variable Screening, Feature Engineering, and Feature Ranking at their defaults.
9. Click Run to execute the flow. Save the flow to the VST folder as Auto Feature Engineer.flw. When the run is complete, maximize the results window by clicking the three dots on the right side of the work area, choose Open in a browser tab, then choose Results.

Engineered features have been created automatically using appropriate feature transformation and extraction techniques. The idea was to automatically learn a set of features (from potentially noisy, raw data) that can be useful in machine learning tasks without needing manually created, handcrafted engineered features.

In the results window, turn your focus to the Label column. What is so special about this Label column? This is the column that holds the description of the newly generated features.

Note: Other columns in the results window are not discussed here. However, they are covered in detail in other SAS courses.

Going forward, you might want to use these generated features for model building. However, the intention of this demo was to explore the automated machine learning capabilities of the tool.

## Lesson 02

## Statistics You Need to Know for Machine Learning
Lesson 02, Section 1 Demo: Listing Observations

TopicTitle

## Demo: Listing Observations (Self-Paced)

In the previous lesson, you loaded all the course data into memory. In this demonstration, you use a SAS Studio flow to list the first 10 observations in the PVA_DONORS data set.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button. Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.
3. In the work area, click on the Table step. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK.
4. Under SAS Steps, expand Examine Data, then double click on List Data.
5. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the List Data node (the input port).Now List Data is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.
6. Click on the List Data node, then click on theOptions tab. Scroll down to Rows to list field and use the dropdown arrow to select First n rows. Leave the default value of Number of rows set to 10 rows.
7. Use the save button to save the flow to the VST folder as Explore PVA.flw. Click Run to run the flow. To view the results, click on Submitted Code and Results, then click Results.

Alternatively, click the three vertical dots on the upper right corner of the open tab, then choose Open in a browser tab, then Results, to open the results in a new browser tab.

- Use the double arrows in the upper right of the navigation pane to hide the navigation pane and maximize the results. The results appear in the Work area. To increase the size of the work area, locate the List Data step settings in the lower half of the work area. Click the Minimize preview button (a minus sign icon) at the top right corner of this section.
- The successful submission of the List Data step produces a list of the first 10 observations from the PVA_DONORS data set. Listing data provides a quick snapshot of your analysis table. By default, the output of the List Data step shows variable labels. You can easily identify the gift-related inputs, such as Gift Count 36 Months, Gift Amount Last, and so on, that display the previous donation history. Scrolling to the right, you can see the demographic inputs, such as Demographic Cluster, Age, and Gender, that provide demographic characteristics about the donors. And last but not the least, there are the two target variables, Target Gift Flag (Response) and Target Gift Amount (Donation_Amt). The binary variable Target Gift Flag represents the response to the campaign and has two levels, Yes and No. The continuous variable Target Gift Amount represents the donation amount from the individuals who have responded to the campaign.
- You will use both target variables in this course, but only one at a time.

## Statistics You Need to Know for Machine Learning
Lesson 02, Section 2 Demo: Exploring Data

## Demo: Exploring Data (Self-Paced)

This demonstration illustrates performing initial data exploration in three parts.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries,LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Part 1: Obtaining Summary Statistics for Continuous Variables

In this demonstration, you perform initial data exploration using a flow with the Summary Statistics node. You obtain summary statistics for continuous variables.

1. Open the previously saved flow Explore PVA.flw from the VST folder.
2. Under SAS Steps, expand the Statistics group, then double click on the Summary Statistics step.
3. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Summary Statistics node (the input port). Now Summary Statistics node is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.
4. Click on the Summary Statistics node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Click the box next to Name to select all the variables. Click OK.
5. Click on the Option tab. The Mean, Standard deviation, Minimum value, Maximum value, and Number of observations are selected by default. Keep these options and additionally select Number of missing values.
6. Still under the options tab, expand Additional Statistics. Select Mode, Skewness, and Kurtosis. Expand Percentiles, then check off Lower quartile, Median, and Upper quartile.
7. Save the flow then right-click on the Summary Statistics node and choose Run node. Open the Results in a new browser tab.

- Notice that several observations have missing entries for the variables GiftAvgCard36 (Gift Amount Average Card 36 Months), DemAge (Age), and Donation_Amt (Target Gift Amount). The Donation_Amt variable has as many as 50% (4843 out of 9686) of its values missing. Also, skewness statistics indicate that most of the variables with the prefix Gift are positively skewed with skewness values ranging from approximately 1.1 to 14.
- Then you examine the DemMedIncome (Median Income Region) variable. Note that it has a minimum value of zero and a very large standard deviation (28707.49). This looks suspicious and warrants further scrutiny. A closer look at its mode (also zero), which is significantly different from its mean and median, reveals an important insight. That is, out of all the income values, zero is the most frequent. It appears that these are the cases for which an income is unknown and has been coded as zero (0). Clearly, having zero income is considerably different from having an unknown income.
- If you properly use the income variable in a predictive model, the solution to this situation is to recode these values as missing and then impute. This solution is presented in upcoming lessons.
- Notice the StatusCatStarAll (Status Category Star All Months) variable, which has a minimum value of zero and a maximum value of 1. In fact, it's a binary variable. By default, SAS treats every numeric variable as interval and every character variable as nominal. Going forward, we will ensure that we use this variable as a categorical variable rather than an interval variable.
- Among the other variables, let's examine the summary statistics of the interval-valued target variable Donation_Amt (Target Gift Amount). You see that the mean (101.64), median (102.6), and mode (92.10) are not very different from one another, with a skewness value (-0.086) being close to zero. These statistics give us the impression that the distribution of the Donation_Amt variable is fairly symmetric. However, it's always recommended to examine the distribution plots of variables to graphically assess the shape and spread of the data. Data satisfying normal distribution is beneficial for model building and is required in upcoming lessons on regression modeling. Next, you examine the distribution of continuous variables.

Part 2: Distribution Analysis of Continuous Variables

In this demonstration, you use a flow with the Distribution Analysis node to examine the distribution of continuous variables.

1. Return to the previous flow containing the Summary Statistics step (Explore PVA.flw). From the Steps menu, under SAS Steps, expand Statistics, then double click on Distribution Analysis.
2. Move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Distribution Analysis node (the input port). Now Distribution Analysis is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.
3. Click on the Distribution Analysis node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Add all the variables except for StatusCatStarAll. A quick way of getting this done is by clicking the select all box next to Name and then deselecting StatusCatStarAll. Click OK to close the Column Selection window.
4. Click the Options tab. Verify that Histogram has been selected. Select the Add normal curve box under Exploring Data to include a normal curve for the histogram. To include specific summary statistics, select the Add inset statistics option and expand the Inset Statistics properties and select Mean, Standard deviation, Skewness, and Kurtosis in addition to the default of Number of observations. To assess normality using the normal probability plot, select the Normal probability plot box under Checking for Normality.
5. Right click on the Distribution Analysis node and choose Run node. Save the file Explore PVA.flw. When the run is complete, you can view the results on a browser tab. Click the More options on the right side of the screen (at the top right side of the open tab). Click Open in a browser tab and then click Results. View the results.

Note: If you see a warning message in the results window that says The maximum size limit has been exceeded, you might want to click the Open in new tab link to view the results. Alternatively, go to the SAS Studio Options menu, and select Preferences. Under SAS Programs, click Results. Under Display warning if results are larger than specific size, change the Maximum result size (MB) from 3 to 10.

Your results window displays histograms for all the analysis variables assigned in the Analysis variables property. In a practical scenario, it is worth spending time to examine the shape and spread of all the analysis variables in data. However, this is overwhelming in the class setup. So, let's examine the distribution of a few chosen variables. Consider the histogram for the Gift Count 36 Months variable.

The histogram and the normal curve superimposed on it indicate that the distribution is positively skewed, and the same can be ascertained from the statistics in the inset box. Scroll further down to examine the distribution of the DemMedIncome variable.

You clearly see a spike around the zero value, which accounts for approximately 25% of the income values. This resonates with the summary statistics, which indicate its mode value to be zero.

Now you consider examining the distribution of our interval-valued target variable: Donation_Amt (Target gift amount).

The histogram superimposed with the normal curve indicates that the variable is relatively normally distributed.

Next, to examine the normal probability plot for Donation_Amt, scroll down further until the bottom of page, and you see that the distribution of data points follows the normal reference line quite closely.

Thus, the data distribution looks fairly normal. This is an important insight about the interval valued target variable which could be useful in upcoming lessons on modeling.

Part 3: Generating Tables and Plots for Categorical Variables

In this demonstration, you use a flow with the One-Way Frequencies node to generate frequency tables and frequency plots from your data.

1. Return to the previous flow Explore PVA.flw. From the Steps menu, under SAS Steps, expand Statistics, then double click on One-Way Frequencies.
2. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the One-Way Frequencies node (the input port). Now One-Way Frequencies is connected to the PVA_Donors data. Click the Arrange nodes button to visually organize the nodes within the flow.
3. Click on the One-Way Frequencies node, then select the Data tab. Next to Analysis variables, click the plus sign to add variables. Add the variables StatusCat96NK, StatusCatStarAll, DemCluster, DemGender, DemHomeOwner, and Response variables.
4. No changes are needed in the Options tab.
5. Save the flow. Click right click on the One-Way Frequencies node and click Run node. Click on Submitted Code and Results, then click Results to view the report. Alternatively, click the More options on the right side of the screen (three vertical dots at the top right side of the open tab), then click Open in a browser tab, and finally click Results.

The flow displays the frequency table and bar charts for the requested variables. Note that the DemCluster variable has as many as 54 distinct levels. Do you see any challenges involving using a categorical variable that has as many as 54 levels as input for model building? Scroll further to examine the distribution of yet another target variable in our data set, Response. The frequency table and distribution plot clearly indicate that the data set has an equal proportion of responses and non-responses.

Do you see any relationship between proportion of non-responses (in the Response variable) and the number of missing values in Donation_Amt?

## Statistics You Need to Know for Machine Learning
Lesson 02, Section 3 Demo: Testing a Hypothesis Using a One-Sample t-Test (Self-Paced)

## Demo: Testing a Hypothesis Using a One-Sample t-Test (Self-Paced)

In this demonstration, you use a flow with the t-Tests node to test whether the population mean donation amount is $100 given the information in the PVA_DONORS table.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.
3. In the work area, click on the Table step. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK. Note that it is also possible to copy the PVA_DONORS node from a previously created flow and paste it into the new flow.
4. Under SAS Steps, expand Statistics, then double click on t-Tests.
5. Next, move your mouse towards the edge of the PVA_Donors node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the t-Tests node (the input port). Now t-Tests is connected to the PVA_Donors data.
6. Click on the t-Tests node, then select the Data tab. Verify that One-sample test is selected under the t-test field.
7. Click the plus sign for the Analysis variable field. Assign Donation_Amt as the analysis variable. Click OK to close the Column Selection window.
8. On the Options tab, in the Alternative hypothesis: mu ≠ field, enter 100.
9. Under Normality Assumption, clear the option to conduct Tests for normality.
10. To select plots, expand the Plots property and use the drop-down menu to choose the option Selected plots. Select the option to display a Confidence interval plot in addition to the default plots of histogram, a box plot, and a normality plot.
11. Save the flow to the VST folder as ttest.flw.
12. Run the flow and examine the results.

The mean donation amount is $101.60, and the associated t statistic value is 4.44. The p-value is <0.0001. Assuming a significance level of 5% (that is, alpha=0.05), you find that the p-value is less than alpha. Thus, you reject the null hypothesis and conclude that the hypothesized population mean donation amount of $100 is significantly different from the sample mean of $101.60. Scroll further to examine the confidence interval plot. It is a horizontal line ranging from 100.9 to 102.4 to graphically represent the 95% confidence interval around the mean estimate of the donation amount.

Notice the vertical reference line drawn at 100 (the hypothesized value). This reference line lies outside the confidence limit, which clearly indicates that the mean value in the sample is statistically different from $100 at an alpha level of 0.05.

Next, the histogram and the q-q plot for donation amount do not show any severe departures from normality although there are some potential outliers present in the data. Therefore, the Student's t test is valid.

At this point, your brain might be struggling to accept whether a difference of $1.60 between the hypothesized value and the sample mean is significant when the Donation_Amt variable ranges from $0 to $211.9. Well, the interpretation that you just made from the t-test is based on the data in hand and the significance level chosen. Recall our discussion on the over-reliance on the arbitrary cutoff of a 5% significance level and the sample size. In inferential statistics the 5% level of significance might work well given the sample size. However, in a big data context, it is not always appropriate. Large sample sizes result in smaller p-values, and in that case, you tend to reject the null hypothesis almost every time. Thus, you better select a very small significance level cutoff when using the p-value approach in the big data world. In this example, we have a significantly large data set (from a statistical perspective) with 4843 observations. Thus, using a 5% level of significance is highly likely to produce some statistically significant results.

Let's see the impact of using a smaller significance level cutoff on the inference that we draw and the confidence intervals.

## Changing the Values of the Level of Significance in the t-Test

13. Navigate to the VST course folder and double click on the SAS program file Ttest change alpha.sas.
14. Look at the SAS program to see the specified alpha level for this ttest:

    proc ttest data=VST.PVA_DONORS sides=2 h0=100 plots(only showh0)=
    	(summaryPlot intervalPlot qqplot) alpha=0.000001;
       var Donation_Amt;
    run;

15. Click the + Code to Flow button and add the program to the previously saved ttest flow.

16. Return to ttest.flw. Right click on the ttest change alpha node and click Run node. Save the flow and examine the Results.

    Note that neither the t statistic value nor the corresponding p-values have changed in comparison to previous run results. What changes is the confidence interval for the mean donation amount and the statistical inference.

    Now you see that the 99.9999% confidence limit values for the mean range from 99.83 to 103.4, which clearly includes the hypothesized value within it. The same interpretation can be drawn from the confidence intervals plot. You see that the reference line at $100 is contained within the confidence limit. Thus, you would reach the conclusion that the mean donation amount is not statistically different from $100.

    Note that there is nothing special about considering a significance level of 5% always. It should depend on the problem that you are studying and the sample size.

## Lesson 03

## Statistics You Need to Know for Machine Learning
Lesson 03, Section 1 Demo: Correlation and Linear Regression

TopicTitle

## Demo: Correlation and Linear Regression (Self-Paced)

In this demonstration, you explore relationships between numeric predictors and the response variable Donation_Amt using correlation analysis in a SAS Studio flow. Then the input GiftAvgCard36, which is most correlated with the target, is used in a simple linear regression.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

Correlation analysis

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps tab, expand Data (Input and Output) group, double click on the Table step.
3. In the upper half of the work area, click on the Table step. In the lower half of the work area, under Table Properties tab, click the file cabinet next to the Library field. Under Libraries on the Left, click on the VST library, then select the PVA_DONORS data on the right and click OK.
4. Scroll down through SAS Steps, expand Statistics group, then double click on Correlation Analysis.
5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Correlation Analysis node (the input port). Now Correlation Analysis is connected to the PVA_DONORS data. Click the Arrange nodes button to visually organize the nodes within the flow.
6. Click on the Correlation Analysis node. In the lower half of the work area, under the Data tab, find the Analysis variables field and click the plus sign to add variables. Add all variables except Donation_Amt and StatusCatStarAll. Do this by clicking the Select all box next to Name and then deselecting Donation_Amt and StatusCatStarAll. Click OK.
7. Find the Correlate with field. Use the plus sign to add the variable Donation_Amt. Click OK.
8. Switch tabs to Options. Under Statistics > Statistics to display, select Selected statistics from the drop-down menu. Check the boxes for Display p-values and Order correlations from highest to lowest (in absolute value).
9. Scroll down and expand Plots. Under Type of plot, select Individual scatter plots from the drop-down menu. Check the Include ellipse box and use the default settings of Prediction Ellipse for the ellipse and 95% for the confidence level.
10. Uncheck the Include inset statistics box.
11. Keep Number of Variables to plot set to 5 and change Maximum number of plot points to No limit.
12. Click Run to execute the flow. Click on the Submitted Code and Results tab at the top. The results appear in the work area. To increase the size of the work area, locate the Correlation Analysis step settings in the lower half of the work area. Click the Minimize previewbutton (a minus sign icon) at the top right corner of this section.  Alternatively, you can click on More options > Open in a browser tab > Results to open the results in a new browser tab.

Note: If you see a warning message in the results window that says The maximum size limit has been exceeded, you might want to click the Open in new tab link to view the results. Alternatively, go to the SAS Studio Options menu, and select Preferences. Under SAS Programs, click Results. Under Display warning if results are larger than specific size, change the Maximum result size (MB) from 3 to 10.

Each cell in the Correlation table lists several pieces of information: the variable correlated with the response Donation_Amt, the estimate of the correlation coefficient, the p-value, and the sample size. The p-value is for a test of the null hypothesis that the population correlation equals zero. The sample size depends on the number of nonmissing values for only the two variables being correlated. Because of this, the sample size can differ for each correlation.

The correlations have been ranked from strongest to weakest. The strongest correlation with the target is approximately 0.65 for GiftAvgCard36. This positive correlation indicates that as GiftAvgCard36 increases, Donation_Amt tends to increase. It will likely be a good predictor in a regression model for Donation_Amt. The weakest correlation with the target is -0.006 for PromCntCard12. This negative correlation is not statistically different from zero at any commonly used alpha level.

Correlations measure the linear associations between variables. Scatter plots can be useful for assessing linear relationships and thus the appropriateness of correlation analysis. For large data sets, scatter plots can be challenging to interpret when many dots fall on top of each other and are indistinguishable. Prediction ellipses can help you visualize the strength and direction of a correlation. The plots show 95% prediction ellipses for 5 of the 20 variable pairs. The 95% prediction ellipses show where 95% of the data points are expected to lie if both variables are normally distributed. The sign of the correlation becomes obvious from the orientation of the ellipse. The narrower the ellipse, the stronger the linear association between the variables. None of the five plots show strong nonlinear relationships, indicating that correlation is a reasonable tool for describing relationships between these variables. Some plots show unusual data points such as the plot showing Gift Amount Last. Unusual data points, often called outliers, are discussed in a later lesson.

Simple linear regression

Next, we will use a regression node in a flow to regress Donation_Amt onto GiftAvgCard36, the most correlated variable identified earlier in this demonstration. This will produce parameter estimates and hypothesis tests that can be used to understand relationship between variables, as well as a regression equation that can be used to predict future donations from lapsing donors.

1. Return to the previous flow containing the Correlation Analysis step. Make sure the Flow tab is selected and active before proceeding further. From the Steps menu on the far left pane, ensure that you are on the SAS Steps tab. Expand the Statistics group, then double click on the Linear Regression step to add it to the existing flow. Connect the PVA_DONORS node to the Linear Regression node.
2. Click on the Linear Regression node. In the lower half of the work area, ensure the Data tab is selected. Locate the Dependent variable field, then click the plus (+) sign and add the variable Donation_Amt. Click OK.
3. Next, find the Explanatory variables field. Use the plus sign to add GiftAvgCard36.
4. Continue working into the lower half of the work area to configure the remaining settings for the analysis. Click on the Model tab. Leave the default option for Model selection method (Full model fitted (no selection)) and leave the Include intercept option selected.
5. Click on the Statistics tab, under Details on Estimates, check the box for Confidence limits for parameter estimates. Use the default value of 95% for Confidence level.
6. Click on the Plots tab. Under Plots to display, choose Selected plots. Check the boxes next to Observed by predicted values plot, Diagnostic plots, and Scatter plot with regression line.
7. Right click on the Linear Regression node and choose Run node.
   Note: You can run a subset of the flow by using the following options:
   - Run node - runs only the currently selected node in the flow.
   - Run from node - runs the currently selected node and all nodes that occur downstream from the currently selected node.
   - Run to node - runs all nodes that occur upstream from the currently selected node. The currently selected node is not run.
8. Save the flow to the VST folder as Correlation and Regression.flw.
9. Click the Minimize Preview button (minus sign icon) located at the top-right corner of the lower half of the work area to collapse the section and then view the results.

In the simple linear regression model, Donation_Amt is regressed on GiftAvgCard36. Only 4111 observations are used. We learned in an earlier lesson that Donation_Amt has missing values for approximately half the observations. Any observation with missing values for the predictor or response is dropped from the analysis.

The Analysis of Variance table shows that the model is significant with P < 0.0001. The R-square indicates that about 42% of the variability in Donation_Amt can be explained by GiftAvgCard36. The adjusted R-square equals 0.42, and the mean square error (MSE) is approximately 392. MSE, also called the error variance, is a measure of the variance in the response variable that is not explained by this model. Both the adjusted R-square and MSE can be used for model comparisons, with higher adjusted R-square or lower MSE indicating a better model.

The Parameter Estimates table shows that the estimated intercept equals 75.41694 and the estimated slope for GiftAvgCard36 equals 1.79672. Thus, the regression model for estimated donation amount is

The p-value for the GiftAvgCard36 parameter indicates that the population slope is significantly different from zero. Thus, this predictor can be useful to predict future donation amounts. The slope of 1.80 indicates that for every additional dollar of GiftAvgCard36, the predicted donation amount will increase by $1.80. The 95% confidence interval on the slope for GiftAvgCard36 indicates the parameter estimate is likely between (1.73, 1.86). The intercept is also significantly different than zero, but tests of the intercept are often less relevant than tests of the slope. In this case, the intercept tells us that the average donation equals $75.42 when GiftAvgCard36 equals zero.

The plot of Observed by Predicted values of Donation_Amt shows that predictions match observed donations poorly when observed donations are under $100.

Several diagnostic plots based on residuals (the difference between observed and predicted values of the response) are shown. These can be used to assess whether the model assumptions are met and to detect extreme observations. Both of these topics are discussed later in this lesson.

The Fit plot shows Donation_Amt plotted against GiftAvgCard36. The best-fit regression line (the solid black line) indicates the predicted value of Donation_Amt for specific values of GiftAvgCard36. The thin light gray shaded region around the regression line indicates the 95% confidence interval on the mean response for specific values of the predictor. It is difficult to see here in the picture Fit Plot, but the confidence intervals are narrowest near the mean values of X and Y and widest at the extremes. The dashed lines show the 95% prediction limits. You can be 95% confident that new observations will lie within these prediction limits.

## Statistics You Need to Know for Machine Learning
Lesson 03, Section 2 Demo: Multiple Regression and Model Selection

## Demo: Multiple Regression and Model Selection (Self-Paced)

In this demonstration, you use a SAS program to create a multiple regression model for Donation_Amt using the stepwise model selection method. You use Akaike's information criterion (AIC) to add and remove variables from the model. The program Multiple Regression.sas is added to the previously created flow Correlation and Regression.flw.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Open the previously created flow Correlation and Regression.flw from the VST folder. We will add a program to this flow.
2. Navigate to the VST folder and double click on Multiple Regression.sas. Click on the SAS Program step and view the code. While the majority of the code is beyond the scope of this course, note the highlighted portions. The first highlighted section (param=glm) indicates the GLM coding of categorical variables. This default coding creates one numeric (dummy) variable for each level of a categorical predictor. Effect coding (param=effect) and reference coding (param=ref) are other available options. They create one fewer dummy variable than GLM coding and are used for comparing levels to the mean or to a reference level respectively.
   The second highlighted section (selection=stepwise (select=aic)) shows that the multiple regression model uses stepwise variable selection with variables being selected based on Akaike’s information criterion. Here is the code:

       ods graphics / imagemap=on;

       proc glmselect data=VST.PVA_DONORS outdesign(addinputvars)=Work.reg_design

       plots=(criterionpanel);

       class DemHomeOwner DemGender DemCluster StatusCatStarAll StatusCat96NK/ param=glm;

       model Donation_Amt=GiftCnt36 GiftCntAll GiftCntCard36 GiftCntCardAll GiftAvgLast GiftAvg36 GiftAvgAll GiftAvgCard36 GiftTimeLast GiftTimeFirst PromCnt12 PromCnt36 PromCntAll PromCntCard12 PromCntCard36 PromCntCardAll DemAge DemMedHomeValue DemPctVeterans DemMedIncome DemHomeOwner DemGender DemCluster StatusCatStarAll StatusCat96NK/ showpvalues selection=stepwise (select=aic);

       run;

       proc reg data=Work.reg_design alpha=0.05 plots(only)=(diagnostics residuals dffits

       observedbypredicted);

       ods select ParameterEstimates DiagnosticsPanel ResidualPlot DFFITSPlot

             ObservedByPredicted;

       model Donation_Amt=&_GLSMOD / vif;

       run;

3. Click the + Code to Flow button and add the program to the previously saved Correlation and regression flow. Navigate to the flow tab of Correlation and regression.flw.
4. Right-click on the Multiple regression node and choose Run node. Save the Correlation and regression flow.
5. Click the Submitted Code and Results tab in the upper half of the work area to view the results.

The multiple regression model used 3101 observations due to complete case analysis. The stepwise-AIC model has 26 potential effects and 88 potential parameters. The number of parameters is higher than the number of effects because of the dummy-coding necessary to account for the levels of the categorical predictors.

The stepwise process started with a model containing only the intercept. GiftAvg36 was added at step 1 because it reduced AIC (that is, it improved model fit) more than any other single variable. For steps 2 through 6, adding variables improved AIC more than removing variables. At step 7, a previously entered variable, PromCntCard12, was removed. Then in steps 8–13, variables were added based on producing the greatest reduction in the model AIC.

Model selection stopped at the optimal value of AIC. Any addition or reduction of a single variable from the model at step 13 increased the AIC and made a worse model by this criterion. The Stop Details table shows potential candidate variables to add or remove, both of which increase the AIC, but by a small amount (approximately 0.4).

Fit criteria plots show on the X axis the 13 models corresponding to each step, and on the Y axis one of four fit statistics. Keep in mind that the models assessed during the stepwise selection process are not all possible models, but only a small subset. Of that subset, only fit statistics for the model present at each step are graphed. These graphs show that the final model had the best values of AIC, AICC, and adjusted R-square of any of the 13 models. The best values are indicated by a star. For the SBC, the best model was the model at step 10, which had three fewer inputs than the final AIC model.

The selected model has 11 predictors and is statistically significant (P < 0.0001), meaning at least one predictor in the model has a slope that is significantly different from zero (assuming alpha = 0.05). The R-square indicates that this model explains 54% of the variability in Donation_Amt. The MSE is 302.96 (compare with MSE=392 in the simple linear regression) and the adjusted R-square equals 0.54 (compare with R-square = 0.42 in the simple linear regression). The multiple regression fits the data better than the simple linear regression model. A note indicates that the p-values have not been adjusted for the model selection process. This refers to the p-values being biased too low when used for automated model selection. (For more information about this, see Flom and Cassell 2007).

The Parameter Estimates table shows the estimates for the intercept and the slopes for each input variable. The two categorical variables, StatusCat96NK and StatusCatStarAll, each have multiple parameter estimates, one for each level of the variable except for the reference level. The reference level has a parameter estimate of zero. The other levels have parameters that describe the difference between that specific level and the reference level. For example, the parameter estimate of 2.4977 for StatusCat96NK A indicates a $2.4977 greater average donation amount for members of the A-group than for members of the S- group.

Notice that several of the effects in the model, such as PromCntCardAll and PromCntCard12 are not significant at the alpha=0.05 level. This is because the model selection approach was based on Akaike's information criterion and not significance level. At each step in the model-building process, a variable was either added or removed that improved the AIC (smaller is better) relative to the previous model. At the final step, no further additions or removals of one variable made the AIC smaller. Using other model selection criteria such as SBC, adjusted R-square, or significance level can result in different final models than the stepwise AIC model (not shown). Some of these other models are discussed later in this lesson. The diagnostics plots for this model are discussed in the next demonstration.

## Statistics You Need to Know for Machine Learning
Lesson 03, Section 3 Demo: Model Diagnostics

TopicTitle

## Demo: Model Diagnostics (Self-Paced)

In this demonstration, you assess the assumptions for the previously fit stepwise selection model. In addition, you check for collinearity problems and for outliers and influential observations.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Return to the results of the Multiple_regression node in Correlation and regression.flw (last demonstration). Rerun the flow if necessary.
2. View the results on the Submitted Code and Results tab, focusing on the diagnostic tools and plots.

The second Parameter Estimates table has a column for variance inflation factors (VIF). These represent the strength of collinearity for each effect in the model. Collinearity is not a violation of the model assumptions, but it can result in model instability as well as other problems. A VIF >10 indicates a strong collinearity problem with the model.

PromCntCardAll has the highest VIF of 12.46, and GiftTimeFirst has the second highest VIF of 9.48. Collinearity inflates the variance of parameters relative to a model without collinearity, but for both of these variables, collinearity did not inflate the variances enough to make them nonsignificant. Both variables have p-values <0.0001. One approach to addressing collinearity problems is to remove variables involved in the collinearity, usually starting with the one with the highest VIF.

Predicted and observed values of the target match better than in the previous simple linear regression model. In the multiple regression model, smaller donations under $50 are not predicted well. They tend to be overestimated by the model with predictions generally over $50. In addition, higher predicted values tend to be overestimates of the observed donation amount.

The first column of the Fit Diagnostics plots can be used to assess whether the model assumptions have been met. The assumptions of independence, constant variance, and normality are necessary to be able to trust the p-values, standard errors, and confidence intervals from the model.

The residual by predicted value graph shows a non-random pattern. Predicted values over $175 all correspond to negative residuals. This suggests that the model form might be inadequate. Perhaps higher-order terms such as quadratic or interaction effects could improve the model fit. Residual variability might be getting smaller as the predicted values get larger. In addition, the residuals versus normal quantiles and the histogram of the residuals suggest deviations from a normal distribution. The histogram shows left skewness instead of a symmetric distribution. If statistical inference were the goal for this model, the researcher would likely modify this model to better meet the assumptions of regression.

The Cook's D and DFFITS plots show the presence of at least 1 to 3 influential observations. The three spikes that stand out in the Cook's D plot indicate cases that have a large influence on the entire set of parameter estimates (the intercept and the 15 slopes). The same three cases are also influential according to the DFFITS statistic. Large values of DFFITS that are well separated from other such values indicate cases that have a large influence on the predicted values.

The plots of residuals against each of the inputs tend to show the same pattern as residuals versus predicted values: many negative residuals at higher values of predictors and potentially nonconstant residual variance. These plots might indicate a misspecified model. Transformations of the inputs or adding quadratic terms can reduce the pattern of negative residuals at higher values of the predictors.

The last several scatter plots show two columns of dots. These correspond to dummy variables created for the two categorical predictors in the model.

Further effort could be used to improve the fit of this model, or other new models could be created and compared with the stepwise AIC model. These potential next steps are discussed later in this lesson.

## Lesson 04

## Statistics You Need to Know for Machine Learning
Lesson 04, Section 2 Demo: Examining Categorical Association

TopicTitle

## Demo: Examining Categorical Association (Self-Paced)

In this demonstration, you use the Table Analysis node in a flow to test an association between the two categorical variables, Response and StatusCatStarAll. You also calculate measures of association such as Cramer's V statistic and the odds ratio to determine the strength of association.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.
3. In the work area, click on the Table step. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK.
4. Under SAS Steps, expand Statistics, then double click on the Table Analysis step.
5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Table Analysis node (the input port). Now Table Analysis is connected to the PVA_DONORS data. Click the Arrange nodes button to visually organize the nodes within the flow.
6. Click on the Table Analysis node, then select the Data tab.
7. Under Row variables, click the plus sign and select the StatusCatStarAll variable. Then click OK to close the Column Selection window.
8. Again, click the plus sign next to Column variables and select the Response variable. Then click OK to close the Column Selection window.
9. On the Cell Statistics tab, under Frequencies, select Expected in addition to Observed.
10. Under Percentages, check the box for Row.
11. Under the Association Statistics tab, under Tests of Association, ensure that Chi-square statistics (including Pearson, likelihood ratio and Mantel-Haenszel…) is selected
12. Scroll down to Measures of Association and select Odds ratio and relative risk (for 2x2 tables).
13. On the Computation Options tab, select the option to Suppress plots.
14. Run the flow and save it to the VST folder as Categorical data analysis.flw.
15. Examine the results. The Results window displays the frequency table, which includes the cell frequency, expected frequency, and row percentage.

Examine the row frequency percentages for combinations of values of the two variables. You notice that the row percentages are different in each column, indicating a possible association between the two variables (Status Category Star All Months and Response). To assess whether the differences between the percentages of response to the campaign across the levels of the status category is greater than would be expected by chance, you need to evaluate the chi-square test results.

Because the p-value for the chi-square statistic is <.0001, which is below 0.05, you reject the null hypothesis at the 5% level and conclude that there is evidence of an association between StatusCatStarAll and Response. The Cramer's V statistic value of 0.1106 indicates that the association detected by the chi-square test is relatively weak. The Odds Ratio and Relative Risks table can also be used to measure the strength of association.

The odds ratio is shown in the first row of the table, along with the 95% confidence limits. To interpret the odds ratio, refer to the contingency table at the beginning of the output. The top row (StatusCatStarAll=0, in this case) is the numerator of the ratio, and the bottom row (StatusCatStarAll=1) is the denominator. The interpretation is stated in relation to the left column of the contingency table (Response=No). The value of 1.56 indicates that an individual with a StatusCatStarAll=0 value has about 1.56 times the odds of not responding to the campaign compared to an individual with StatusCatStarAll=1. This is equivalent to saying that an individual with StatusCatStarAll=1 has 1.56 times the odds of responding to the campaign compared with an individual with StatusCatStarAll=0.

Relative risk estimates for each column are interpreted as probability ratios rather than odds ratios. You get a choice of assessing probabilities of the left column (Column 1) or the right column (Column 2). For example, the Column 1 relative risk shows the ratio of the probabilities of StatusCatStarAll=0 to StatusCatStarAll=1 being in the left column (56 / 44.9 = 1.2472).

## Statistics You Need to Know for Machine Learning
Lesson 04, Section 3 Demo: Fitting a Multiple Logistic Regression Model

## Demo: Fitting a Multiple Logistic Regression Model (Self-Paced)

In this demonstration, you fit a multiple logistic regression model using the Logistic Regression.sas program in a flow. The program uses Response as the target variable and GiftCnt36, DemMedHomeValue, GiftTimeLast, and StatusCatStarAll as the predictor variables. The goal is to predict whether a customer is likely to donate to the donation campaign.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Open the previously created flow Categorical Data Analysis.flw from the VST folder. We will add a program to this flow.
2. Still in the VST folder, double click on Logistic Regression.sas. View the code.
3. The SAS Program step contains the following code. Note the oddsratio statement to calculate odds ratios comparing groups with a one-unit difference. Two other statements start with an asterisk, which prevents the code from being executed:

     proc logselect data=VST.PVA_DONORS association;
      partition fraction(validate=0.5 seed=12345);
      class StatusCatStarAll;
      model Response(event='Yes')=StatusCatStarAll GiftCnt36 GiftTimeLast
      DemMedHomeValue / link=logit;
      code file="/home/student/workshop/VST/logistic_score.sas";
      oddsratio;
      *ods select oddsratios;
      *oddsratio StatusCatStarAll GiftCnt36 GiftTimeLast DemMedHomeValue/
        unit (DemMedHomeValue=1000);
    run;

4. Click the + Code to Flow button and add the program to the previously saved Categorical Data Analysis flow. Navigate to the flow tab of Categorical Data Anlysis.flw.
5. Right-click on the Logistic Regression node and select Run node. Save the Categorical Data Analysis flow.
6. Examine the results from the Logistic Regression node. If you want to view the results on a separate browser tab, click the More options (three dots) menu and select Open in a browser tab > Results.

On the Results tab, the Model Information table displays information about the model, including the data source, distribution, link function, and optimization technique.

The Number of Observations table contains the number of observations that were read from the source data and used in the analysis. Because the data were partitioned, the table displays the number of observations in each partition.

The Response Profile table shows the distribution of the target variable values for each partition. A Class Level Information table lists the levels of categorical predictors.

The Global Null Hypothesis Test table provides a likelihood ratio test for the hypothesis of whether the final model provides a better fit than a model without effects (an intercept-only model). The p-value (<0.0001) column in the global null hypothesis test table indicates that this model is statistically significant at the 5% level of significance, and at least one of the predictors in the model is useful in predicting Response..

The Fit Statistics table displays a variety of likelihood-based measures of fit. These statistics are useful for assessing the fit of the model to your data. The statistics are computed for each data role when you partition the data.

The Parameter Estimates table shows the parameter estimates, their standard errors, and the p-values for assessing statistical significance. All parameter estimates are on the logit scale. You can use these parameter estimates to define your logistic regression equation.

Because reference cell coding was used for the classification variable, the effect is measured against the reference level. For the StatusCatStarAll variable, level 1 is used as the reference level. So, the estimate for StatusCatStarAll | 0 shows the difference in logits between individuals with 0 and 1 for StatusCatStarAll. The table shows that the parameter estimates for all the effects are statistically significant at the 5% level of significance.

The Association of Predicted Probabilities and Observed Responses table shows that the concordance index (c) equals 0.5938 for the training data and 0.6104 for the validation data. The c statistic value for this model indicates that approximately 60% of the positive and negative response pairs are correctly sorted using StatusCatStarAll, GiftCnt36, GiftTimeLast, and DemMedHomeValue.

The Odds Ratio Estimates table can be used to establish the relationship between predictors and the odds of the event. It is often easier to report odds ratios by first transforming the decimal value to a percent difference value. In this example, the odds ratio estimate for StatusCatStarAll when transformed to percent difference is (0.820-1)*100= -17.96%. In other words, the odds ratio estimate for StatusCatStarAll shows that, when adjusting for other predictor variables, individuals with zero for StatusCatStarAll had ~18% lower odds of donation than the individuals with 1 for StatusCatStarAll. For GiftCnt36, the odds ratio estimate equals 1.084. This means that for each additional donation in the past 36 months, the odds of donation during the campaign changed by a factor of 1.084, an 8.4% increase.

Note: The unusual value of 1.000 for the DemMedHomeValue odds ratio has a simple explanation. Unit (that is, single dollar) changes in the home value do not change the odds of response by an amount captured in three significant digits. To obtain a more meaningful value for this input's effect on response odds, you can calculate an odds ratio comparing median home values that differ by $1000. We’ll alter the code to do this in the next step.

7. Go back to Categorical Data Analysis flow and click on the Logistic Regression node. You will edit the code to change how the oddsratio for DemMedHomeValue is calculated. Add an asterisk to the first oddsratio statement and remove the 2 asterisks from the ods select statement and from second oddsratio statement (which is followed by 4 variable names). Now the oddsratio statement will calculate an odds ratio for DemMedHomeValue, comparing groups with a $1000 difference. The code should appear as follows:

    proc logselect data=VST.PVA_DONORS association;
      partition fraction(validate=0.5 seed=12345);
      class StatusCatStarAll;
      model Response(event='Yes')=StatusCatStarAll GiftCnt36 GiftTimeLast
      DemMedHomeValue / link=logit;
      code file="/home/student/workshop/VST/logistic_score.sas";
      *oddsratio;
      ods select oddsratios;
      oddsratio StatusCatStarAll GiftCnt36 GiftTimeLast DemMedHomeValue/
        unit (DemMedHomeValue=1000);
       run;

8. Run the Logistic Regression node again and view the results.

Now the odds ratio for DemMedHomeValue shows a value of 1.001 with a 95% confidence interval of (1.001, 1.002). This indicates that for every $1000 increase in DemMedHomeValue, the odds of donating increase by 0.12%.

## Statistics You Need to Know for Machine Learning
Lesson 04, Section 4 Demo: Scoring a Logistic Regression Model

TopicTitle

## Demo: Scoring a Logistic Regression Model (Self-Paced)

In this demonstration, you score a new data set using the score code from the logistic regression model that you created earlier. This enables you to add predictions from a trained model to a score data set.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Open the previously created flow Categorical Data Analysis.flw from the VST folder. We will add a program to this flow.
Still in the VST folder, double click on Scoring Logistic.sas and view the code.
2. The SAS Program contains the following code. It uses the score code from the logistic regression model to score the SCORE_PVA data set.
3. The predictions are saved to the PVAScored data set:

    data VST.PVAScored;
      set VST.SCORE_PVA;
      %include "/home/student/workshop/VST/logistic_score.sas";
    run;

    proc contents data=VST.PVAScored;
    run;

4. Click the + Code to Flow button and add the program to the previously saved Categorical Data Analysis flow. Navigate to the flow tab of Categorical Data Anlysis.flw.
5. Right-click on the Scoring Logistic node and select Run node. Save the Categorical Data Analysis flow.
6. Examine the Results as well as the Output Data tab. Note the variables P_response and I_response in both the Results and the Output Data tabs

Artial view of the output table (PVASCORED) is shown below:

The P_Response column shows the predicted probability of the target Response=Yes. The I_Response column is an indicator variable that takes the value of Yes when the value of P_Response is greater than or equal to 0.5 and No otherwise.

Note: Your output might show a different listing of scored data.

## Lesson 05

## Statistics You Need to Know for Machine Learning
Lesson 05, Section 2 Demo: Modifying and Correcting Data

TopicTitle

## Demo: Modifying and Correcting Data (Self-Paced)

In this demonstration, you use a flow with the t-Tests node to test whether the population mean donation amount is $100 given the information in the PVA_DONORS table.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. In the work area, click the plus sign next to the open tabs to create a new tab. From the drop-down menu, choose Flow. Alternatively, select Flow from the New menu at the top left of SAS Studio.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, Data (Input and Output), double click on Table.
3. In the work area, click on the Table node. At the bottom, under Table Properties, click the file cabinet next to the Library field. Click on VST under Libraries, then select the PVA_DONORS data on the right and click OK.
4. Under SAS Steps, expand the Transform Data group, then double click on the Recode Values step. Right-click the Recode Values node and choose Add output port.
5. Next, move your mouse towards the edge of the PVA_DONORS node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS node to the box on the left side of the Recode Values node (the input port). Now Recode Values is connected to the PVA_DONORS data.
6. Add a second Table node to the flow. For the second Table, enter VST for Library and MISSING_PVA for Table name.
7. Now connect the box on the right side of the Recode Values node (the output port) to the MISSING_PVA node. Click the Arrange Nodes button to visually organize the flow.

8. Click on the Recode Values node. Under the Data tab, the Numeric variable radio button is selected by default. Click on the plus sign (+) next to Variable to recode. Choose DemMedIncome and click OK.
9. Click on the Values tab. Under Recode values, enter zero (0) in the Old value field. In the New value field, add a single period (.) which indicates missing values.
10. Click on the Output tab. In the Recoded variable name field, change the default name to Rep_DemMedIncome. Ensure that Write to another data set and Replace existing output table are both selected.

Run the flow. Save the flow in the VST folder as Data Preprocessing.flw.

11. After the flow runs successfully, click the Submitted Code and Results tab. Click the Output Data tab to view the Missing_PVA data set and scroll to the right. You see that a new column, Rep_DemMedIncome, has been created that has missing values against the zero values present in the DemMedIncome column.

## Statistics You Need to Know for Machine Learning
Lesson 05, Section 2 Demo: Managing Missing Values

TopicTitle

## Demo: Managing Missing Values (Self-Paced)

In this demonstration, you learn to address missing values in the MISSING_PVA data set. The data set has quite a few variables with missing values. However, for brevity, let's focus on the Rep_DemMedIncome variable. Use the Imputation node in a flow to impute synthetic data values.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Open the previously created flow Data Preprocessing.flw from the VST folder.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, expand Prepare Data, then double click on the Imputation step.
3. Right click on the Imputation step and choose Add output port.
4. Next, move your mouse towards the edge of the MISSING_PVA node until the pointer becomes a hand icon. Click and drag an arrow from MISSING_PVA node to the box on the left side of the Imputation node (the input port). Now Imputation is connected to the MISSING_PVA data.
5. Add another Table step to the flow by double clicking Table under Data (Input and Output). For this node, under Table Properties, enter VST for Library and IMPUTED_PVA in the Table name field.
6. Connect the output port of the Imputation node to the IMPUTED_PVA node. Use the Arrange nodes button to visually organize the nodes in the flow.

7. Click on the Imputation node, then select the Data tab.
8. Under Interval Variables, click the plus sign corresponding to Replace missing values with the mean and select the Rep_DemMedIncome variable. Then click OK to close the Column Selection window.
Note: Users also have the option to impute the missing values in interval variables with the median or a random number. If required, users can choose different methods for different variables.
Note: The Imputation step also supports imputing missing values in nominal variables by mode.

9. Click the Output tab. Make sure Save imputed data and Replace existing output table are both selected.

Note: To expedite the demonstration in the class setup, a CAS table in the VST library has already been saved with the same name. Thus, the Replace option is used.

10. Next, to include all the variables from the input CAS table in the output CAS table, click the All variables radio button under Include variables from the input CAS table.
11. Click Run to execute the flow. Save the file Data Preprocessing.flw. When the run is complete, view the results in the work area. You clearly see that a new input, IM_Rep_DemMedIncome, is added to the data set, and it has missing values replaced by a synthetic value ($53513, the mean in this case) and nonmissing values copied from the original input.

12. Click the Output Data tab to view a snapshot of the output table. Scroll your cursor to the right, and you see the new column, IM_Rep_DemMedIncome, that contains the value 53513 for every missing value in the Rep_DemMedIncome column. Because both the versions of the income variables are available in the data set, going forward, you should use the IM_Rep_DemMedIncome variable for analysis.

## Statistics You Need to Know for Machine Learning
Lesson 05, Section 2 Demo: Transforming Inputs

TopicTitle

## Demo: Transforming Inputs (Self-Paced)

Transforming inputs can sometimes improve the fit of a model to the data. Recall that in our data set we have several gift-related variables that are heavily skewed. In this demonstration, you learn to transform such variables using a pre-created flow to regularize their distribution. For brevity, we will consider just one of the gift-related variables in this demonstration.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Open the previously created flow Data Preprocessing.flw from the VST folder.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, expand Prepare Data, then double click on the Transform columns step.
3. Next, move your mouse towards the edge of the IMPUTED_PVA node until the pointer becomes a hand icon. Click and drag an arrow from IMPUTED_PVA node to the box on the left side of the Transform Columns node (the input port). Now Transform Columns is connected to the IMPUTED_PVA data.
4. Add another Table step to the flow by double clicking Table under Data (Input and Output). For this node, under Table Properties, enter VST for Library and PVA_DONORS_FINAL in the Table name field.
5. In the work area, connect the output port of the Transform Columns node to the PVA_FINAL_DONORS node. Click the Arrange nodes button to visually organize the flow.
6. Click on the Transform Columns node. Under the Data tab, Find Select a server for this step and choose the CAS radio button.
7. Scroll down to the Variable 1 field under Transform 1. Click the plus sign (+) and choose GiftCnt36. Click OK to close the Column Selection window.
8. In the Transform field under GiftCnt36, use the drop-down arrow to choose Specify custom transformation. In the Custom transformation field, type in log(GiftCnt36+1).

Note the +1 added to the variable GiftCnt36. Some donors did not donate in the past 36 months, so have a GiftCnt36 value of zero. The log of zero is undefined, so adding 1 to GiftCnt36 prevents the transformation from generating missing values when GiftCnt36=0.

9. Click on the Output tab. Make sure Replace existing output table is selected. Under Specify the data to show in the Results tab, use the dropdown menu to select Specific number of observations. Leave the default number of observations set to 10.
10. Run the flow and save Data Preprocessing.flw. View the Results.

Partial view of PVA_DONORS_FINAL:

The Results tab in the work area shows the first ten rows of PVA_FINAL_DONORS. Note that you may see a different 10 rows on your computer. You can see that a new input named tr1_GiftCnt36 has been added to the table.

Going forward you might want to use this log transformed variable over the original variable.

## Statistics You Need to Know for Machine Learning
Lesson 05, Section 2 Demo: Performing Feature Selection

TopicTitle

## Demo: Performing Feature Selection (Self-Paced)

Removing redundant and irrelevant inputs from a training data set often improves the performance of a model. In this demonstration, you use a pre-created flow to select potentially useful features (variables).

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Open the previously created flow Data Preprocessing.flw from the VST folder.
2. Click on the Steps icon on the left side of the Navigation Pane. Under SAS Steps, expand Prepare Data, then double click on the Variable Selection step.
3. Next, move your mouse towards the edge of the PVA_DONORS_FINAL node until the pointer becomes a hand icon. Click and drag an arrow from PVA_DONORS_FINAL node to the box on the left side of the Variable Selection node (the input port). Now Variable Selection is connected to the PVA_DONORS_FINAL data. Click the Arrange nodes button to visually organize the flow.
4. Click on the Variable Selection node. Under the Data tab, ensure that the Unsupervised variable selection radio button has been selected.
5. Expand Explanatory Variables using the dropdown arrow. Under Explanatory Variables, find the field for Continuous variables and click the plus sign (+) to add numeric columns to the variable list. Select all the numeric inputs except GiftCnt36, StatusCatStarAll, DemMedIncome, Donation_Amt, and Rep_DemMedIncome. You can do so by clicking the Select all check box next to the Name field to include all the numeric inputs and then clear the boxes corresponding to GiftCnt36, StatusCatStarAll, DemMedIncome, Donation_Amt, and Rep_DemMedIncome. Click OK to close the Column Selection window.
6. Scroll down to the Classification variables field. Click the plus sign (+) to add variables. In the Column Selection window, select StatusCat96NK, StatusCatStarAll, DemGender, and DemHomeOwner. Click OK to close the Column Selection window.
7. Click on the Model tab to view these settings. Leave the default option Main effects model selected.
8. Next, click on the Options tab. Leave the default settings for Select variables based on: Correlations, Proportion of total variance to be explained: Use 0.9, and Maximum number of selected effects: Use 200.
9. Scroll down and select the box next to the Specify minimum increment of explained variance property. Verify that the Increment of explained variance value is set at 0.001.
10. Finally, under the Plots option, select the Iteration plot box to include an iteration plot in the results.
11. Typically, an output port connected to a table node would be added to Variable Selection to save the variable list to a data set. In the interest of time, here we will just right-click on the Variable Selection node and choose Run node to view the variable list in the Results. Save Data Preprocessing.flw and view the results.

When the run is complete, the results are displayed in the work area. Note that the variable reduction algorithm has used 5908 observations out of a total of 9686 observations. This is because of the presence of several missing values. Remember that, for brevity, you imputed the missing values in just one of the several variables. In practice, you should impute the missing values in all the variables before performing variable selection.

The Selection Summary table lists the variables in the order of selection. The variable that explains most of the variability in the input space is selected first. In the current example, Promotion Count All (PromCntAll) explains around 20% of variability, which is the highest among all inputs. The selection process stops after 17 iterations. Can you guess why? Well, the final parameter added to the selection summary resulted in just over 90% of the original variability being explained, triggering the algorithm to stop.

Next you see the Selected Effects table, which provides a summary of selected variables. These are the 12 variables that you might want to use as features in any supervised machine learning algorithm that follows.

When you scroll down farther in the results window, you see the iteration plot that summarizes the incremental variance explained by adding a variable in each iteration. This is essentially a grouped bar chart of increasing size in which blue bars represent the base value of variance explained and yellow bars represent the incremental value.

## Statistics You Need to Know for Machine Learning
Lesson 05, Section 3 Demo: Running a Neural Network Model and Tuning Its Hyperparameters

TopicTitle

## Demo: Running a Neural Network Model and Tuning Its Hyperparameters (Self-Paced)

In this demonstration, you use a supervised machine learning algorithm to train a neural network a model. Using a pre-created flow, you also partition the data to perform honest assessment. Finally, you tune the hyperparameter values to improve the performance of the model.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

## Demo Steps

1. Navigate to the VST folder and double click on Neural_Network_PVA_Final.flw.
2. Click on the SAS Program step and view the code. While the majority of the code is beyond the scope of this course, note the highlighted portions. The partition statement partitions the data into 50% for training and 50% for validation. The hidden statement creates 50 hidden units in the neural network. The REGL2=0.1 option in the OPTIMIZATION statement is used to apply an L2 (ridge) regularization. This adds a penalty to the sum of the squared slopes, which helps prevent overfitting by discouraging overly complex models. Here is the code:

    proc nnet data=VST.PVA_DONORS_FINAL standardize=std;
    	partition fraction (validate=0.5 seed=12345);
    	target Response / level=nominal;
    	input GiftAvg36 PromCntAll PromCnt36 / level=interval;
    	input StatusCat96NK DemGender DemHomeOwner / level=nominal;
    	hidden 50;
    	train stagnation=5 outmodel=VST.Nnet_model;
    	optimization regL2=0.1;
    	*train stagnation=5 outmodel=VST.Nnet_model2;
    	*optimization regL2=0.001;
    run;

In a practical scenario, you should consider all 12 features selected in the previous demo performing variable selection, but for the purposes of brevity, here we will be using only the top 6 features (inputs).

3. Click Run to execute the flow and open the results in a new browser tab.

The Model Information table includes information about the number of observations used to train the model, the number of hidden layers, hidden nodes, and the number of parameters estimated.

Note that the misclassification rate for validation data is 0.4511. Because the target variable is nominal, the algorithm tries to choose parameter estimates that can optimize the estimation criterion, which is the misclassification rate in this case.

The Iteration History table indicates that the algorithm performed 9 iterations in all, with the final model having the smallest validate error value (0.4511).

Note: Some SAS Visual Data Mining and Machine Learning models (including the neural network) are created using a nondeterministic process. This means that you might experience different results when you rerun the task.

To improve the performance of the model, you might want to add more inputs or tune the hyperparameter values, or both.

Next you will edit some of the hyperparameters.

4. Return to the flow and update the L2 hyperparameter from 0.1 to 0.001. To make this change, place an asterisk (*) before the first OPTIMIZATION statement and remove the asterisk from the second OPTIMIZATION statement. Do the same for the TRAIN statements: place an asterisk before the first statement and remove the asterisk from the second statement. This change will save the updated neural network as Nnet_model2 in the VST library. The code will now appear as follows:

    proc nnet data=VST.PVA_DONORS_FINAL standardize=std;
    	partition fraction (validate=0.5 seed=12345); target Response / level=nominal;
    	input GiftAvg36 PromCntAll PromCnt36 / level=interval;
    	input StatusCat96NK DemGender DemHomeOwner / level=nominal;
    	hidden 50;
    	*train stagnation=5 outmodel=Nnet_model;
    	*optimization regL2=0.1;
    	train stagnation=5 outmodel=VST.Nnet_model2;
    	optimization regL2=0.001;
    run;

5. Click Run to execute the flow and view the Results.

The Model information table indicates that the chosen model is the one with a misclassification rate on validation data equal to 0.4317, which is slightly lower than that of the previous model.

The Iteration History table indicates that the algorithm has performed as many as 103 iterations in the search for an optimal solution. However, the optimal solution was obtained at iteration number 53, producing the smallest misclassification rate on the validation data.

Note that the result produced by the neural network is not as interpretable as the regression models you saw earlier. This is often cited as one of the limitations of machine learning models, a topic that is discussed in the next section.