Repository navigation
Expand file tree
/
Copy pathFeatureEngRegularization.m
More file actions
1429 lines (1274 loc) · 103 KB
/
Copy pathFeatureEngRegularization.m
File metadata and controls
1429 lines (1274 loc) · 103 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
%[text] # Feature Engineering and Regularization
%[text] [⇦ Main Menu](file:../MainMenu.m)
%[text] Feature engineering and regularization help improve regression models by refining how inputs are represented and how model complexity is controlled. In this lesson, you will transform and construct features, compare feature sets, select useful predictors, and apply Lasso and Ridge regression to improve generalization.
%[text]{"align":"center"} 
%[text:tableOfContents]{"heading":"Table of Contents"}
%[text] **Before you get started:**
%[text] %[text:anchor:H_BF646A47] This live script is intended to be used with the code hidden. On the **View** tab of the MATLAB toolstrip, in the **View** section, select **Hide Code**. Alternately, select **Hide Code** using the icon  at the top right of the Live Editor pane.
%[text]  Although the code is hidden, some interactivity requires familiarity with MATLAB. If you need more instruction, consider taking [MATLAB Onramp](https://matlabacademy.mathworks.com/details/matlab-onramp/gettingstarted), a free 2 hour online tutorial that teaches the essentials of MATLAB.
%[text] %[text:anchor:H_8F8A032D]  For an optimal experience, follow the instructions and steps in the given sequence. Proceed to a new section only after completing the preceding one. Some sections depend on variables created in prior sections and will generate errors if run out of order.
%[text] The  and  icons refer to two different types of interactive activities that you will find in this script. The  usually indicates an interaction where you will explore the visualization of some concept introduced in this script. The  interactions are designed to challenge your understanding of those concepts and may be used for grading and completion checks by your professor.
%%
%[text] ## Why Engineer Features?
%[text] A regression model can only learn patterns that are represented by its input features. While some relationships are apparent in the original predictors, many real-world problems contain nonlinear trends, periodic behavior, or interactions that a simple linear model cannot capture directly.
%[text] Feature engineering is the overall process of improving how predictors are represented for the model. In this lesson, feature **transformation** means re-expressing one predictor with a mathematical mapping such as $x^2${"altText":"x^2"} or $\\log(x)${"altText":"log(x)"}, while feature **construction** means building a richer set of predictors from the original variables, often including transformed terms and interaction terms together.
%[text]  **Try**. Click **View Example Features** to see a candidate set of features over the range of $x${"altText":"x"}.
%[control:button:210a]{"position":[1,2]}
%%\ Visualize Candidate Feature Transformations
if ~exist("x","var") || isempty(x)
x = linspace(-1,1,400).';
end
x = x(:);
Features = GetFeatureLibrary();
Style = DefaultPlotStyle();
figure(Units="normalized",Position=[0.05 0.05 0.9 0.6])
tiledlayout(2,4,TileSpacing="compact",Padding="compact")
for k = 1:numel(Features)
nexttile
y = Features(k).Fcn(x);
scatter(x,y,Style.DataMarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,SeriesIndex=Style.DataSeriesIndex)
xlabel("$x$",Interpreter=Style.Interpreter)
ylabel(Features(k).Latex,Interpreter=Style.Interpreter)
title(Features(k).DisplayName,Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="tight")
end
%[text] Better features often improve predictive performance, but they also increase model complexity. As models become more flexible, they become more susceptible to overfitting. Later, we will explore regularization methods that help control model complexity while preserving predictive accuracy.
%[text]  **Exercise 1.** Which statement best describes the relationship among feature engineering, feature transformation, and feature construction?
%[text] a. Feature engineering is the umbrella process, and transformation and construction are two distinct ways to engineer features.
%[text] b. Feature transformation mainly improves features that were already engineered.
%[text] c. Feature construction is the same as feature selection.
%[text] d, Feature engineering refers only to interaction terms.
CheckAnswer("ExerciseFeatureHierarchy","Select") %[control:dropdown:2af3]{"position":[40,48]}
%%
%[text] ## Feature Transformations: Building Intuition
%[text] A feature transformation is one type of feature engineering. It replaces a predictor with a mathematically transformed version of the same underlying information. The goal is not to change the regression algorithm, but to provide the algorithm with a representation that better reflects the underlying relationship.
%[text] Before comparing many possible transformations, it is useful to develop intuition by looking at the data. Often, the shape of a scatter plot suggests an appropriate transformation. For example:
%[text] - Curved trends may benefit from polynomial features.
%[text] - Exponential growth may require an exponential basis such as $e^{cx}${"altText":"exp(c\*x)"}. If you transform the response instead, taking $\\log(y)${"altText":"log(y)"} can also linearize the relationship.
%[text] - Periodic behavior may be represented using sine or cosine functions with a period that matches the data. \
%[text] ### 1. Select a 1-D Dataset
%[text]  **Try**. **Select a dataset** from the dropdown menu, then click **Plot Selected Dataset** to visualize the raw pattern.
%[text] **Expected Results**
%[text] Examine the shape of the data before selecting a transformation. Curved patterns often suggest polynomial features, exponential patterns may call for an exponential basis or a transformed response, and repeating patterns may require sine or cosine features with an appropriate period. The goal is to choose a representation that makes the relationship easier for a regression model to learn.
DatasetChoice = "Cubic trend"; %[control:dropdown:a001]{"position":[17,30]}
%[control:button:a002]{"position":[1,2]}
if ~strcmp(DatasetChoice,"Select")
[x,y,datasetLabel] = MakeManualDataset(DatasetChoice);
Style = DefaultPlotStyle();
figure
scatter(x,y,Style.DataMarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,SeriesIndex=Style.DataSeriesIndex)
xlabel("x",Interpreter=Style.Interpreter)
ylabel("y",Interpreter=Style.Interpreter)
title("Selected Dataset: " + datasetLabel,Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
else
warning ("Select a valid option.")
return
end
%[text]  **Pro-tip**. Use the visual pattern to guess which transformation may produce a more linear relationship.
%%
%[text] ### 2. Test a Candidate Transformation
%[text]  **Try**. Choose a transformation from the **Select a feature vector** dropdown menu. If you choose sine or cosine, set the period with the **Select a period** dropdown menu. Then click **Test Selected Transformation** to compare the raw and transformed fits.
Features = GetFeatureLibrary();
FeatureItems = string({Features.DisplayName});
Periods = GetPeriodLibrary();
PeriodItems = string({Periods.Label});
SelectedFeatureChoice = FeatureItems(6); %[control:dropdown:a003]{"position":[25,40]}
SelectedPeriodChoice = PeriodItems(2); %[control:dropdown:8575]{"position":[24,38]}
%[control:button:a004]{"position":[1,2]}
if exist("DatasetChoice","var")
[x,y,~] = MakeManualDataset(DatasetChoice);
SelectedFeature = GetSelectedFeatureSpec(SelectedFeatureChoice,SelectedPeriodChoice);
[~,~,summary] = TestFeature(x,y,SelectedFeature,PlotRaw=false,ShowStats=false);
disp("Selected transformation: " + SelectedFeature.DisplayName)
disp("Model comparison for the selected transformation:")
disp(summary)
else
warning("Select a dataset above.")
return
end
%[text] **Reflect.** Does the transformation make the relationship more linear? Do the residuals look less structured after the transformation?
%[text]  **Exercise 2.** What makes feature transformation different from feature construction in this lesson?
%[text] a. Transformation and construction both mean selecting a smaller subset of predictors.
%[text] b. Transformation re-expresses one predictor, while construction builds a broader feature set from the original variables.
%[text] c. Construction replaces one predictor, while transformation combines several predictors.
%[text] d. There is no meaningful difference between them in this lesson.
CheckAnswer("ExerciseTransformVsConstruct","Select") %[control:dropdown:58ad]{"position":[44,52]}
%%
%[text] ## Comparing Feature Transformations
%[text] Visual inspection helps generate hypotheses, but selecting the best transformation requires objective evaluation. In this section, several candidate transformations are compared using cross-validation to determine which representation is most likely to generalize to new data.
%[text] ### 1. Select a Dataset
%[text]  **Try**. Choose a nonlinear dataset from the **Select a dataset** dropdown menu and click **Plot Dataset** to inspect its raw pattern.
DatasetChoiceCV = "Sinusoidal pattern"; %[control:dropdown:a005]{"position":[19,39]}
%[control:button:a006]{"position":[1,2]}
if ~strcmp(DatasetChoiceCV,"Select")
[x,y,cvLabel] = MakeCVDataset(DatasetChoiceCV);
Style = DefaultPlotStyle();
figure
scatter(x,y,Style.DataMarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,SeriesIndex=Style.DataSeriesIndex)
xlabel("x",Interpreter=Style.Interpreter)
ylabel("y",Interpreter=Style.Interpreter)
title("Selected Dataset: " + cvLabel,Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
else
warning("Select a valid option.")
return
end
%%
%[text] ### 2. Evaluate Candidate Transformations
%[text] A model with the lowest training error is not necessarily the best model. Cross-validation estimates how well each transformation is expected to perform on unseen data and provides a more reliable basis for comparison.
%[text] Include candidates that reflect the patterns seen in the data. Over a limited input range, a polynomial can sometimes mimic exponential or periodic behavior well enough to score competitively, but that does not make it the best representation outside the observed range.
%[text]  **Try**. Click **Evaluate Models** to fit the candidate transformations and compare training and cross-validated performance.
%[control:button:a007]{"position":[1,2]}
%%\ Evaluate Candidate Transformations
if exist("DatasetChoiceCV","var")
[x,y,~] = MakeCVDataset(DatasetChoiceCV);
TransformAnalysis = EvaluateTransformLibrary(x,y);
Results = TransformAnalysis.Results;
Style = DefaultPlotStyle();
figure
b = bar([Results.MSE Results.MSECV],"grouped");
b(1).SeriesIndex = Style.DataSeriesIndex;
b(2).SeriesIndex = Style.FitSeriesIndex;
xticklabels(Results.Transform)
xtickangle(25)
ylabel("MSE",Interpreter=Style.Interpreter)
title("Transformation Comparison: Training Versus Validation Error",Interpreter=Style.Interpreter)
legend("Training MSE","Cross-Validated MSE",Location="northwest",Interpreter=Style.Interpreter)
% Use TeX so labels like x^2 render without LaTeX math wrappers.
ax = gca;
ax.TickLabelInterpreter = "tex";
ApplyDefaultAxesStyle(Style)
disp("Transformation ranking by cross-validated MSE:")
disp(Results)
else
warning("Select a dataset from above.")
return
end
%%
%[text] ### 3. Visualize the Best Transformation
%[text] After identifying the best transformation numerically, examine it graphically in both the original and transformed feature spaces.
%[text]  **Try**. Click **Visualize Best Transformation** to visualize the best-performing transformation in the original space, transformed space, and residual plots.
%[control:button:a008]{"position":[1,2]}
[x,y,~] = MakeCVDataset(DatasetChoiceCV);
TransformAnalysis = EvaluateTransformLibrary(x,y);
Style = DefaultPlotStyle();
xBest = TransformAnalysis.BestFcn(x);
valid = isfinite(xBest) & isfinite(y);
xRaw = x(valid);
yBest = y(valid);
xBest = xBest(valid);
mdlRaw = fitlm(xRaw,yBest);
[xSorted,~] = sort(xRaw);
yPredRaw = predict(mdlRaw,xSorted);
[xBestSorted,~] = sort(xBest);
yPredBest = predict(TransformAnalysis.BestModel,xBestSorted);
figure(Units="normalized",Position=[0.05 0.05 0.9 0.6])
tiledlayout(2,2)
sgtitle("Feature Transformation Diagnostics",FontWeight="bold",Interpreter=Style.Interpreter)
nexttile
scatter(xRaw,yBest,Style.DataMarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,SeriesIndex=Style.DataSeriesIndex)
hold on
plot(xSorted,yPredRaw,LineWidth=2,SeriesIndex=Style.FitSeriesIndex)
hold off
title("Original Space",Interpreter=Style.Interpreter)
xlabel("x",Interpreter=Style.Interpreter)
ylabel("y",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
nexttile
PlotResiduals(mdlRaw,"Original")
nexttile
scatter(xBest,yBest,Style.DataMarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,SeriesIndex=Style.DataSeriesIndex)
hold on
plot(xBestSorted,yPredBest,LineWidth=2,SeriesIndex=Style.FitSeriesIndex)
hold off
BestLatex = string(TransformAnalysis.BestLatex);
title("Transformed Space: " + BestLatex,Interpreter="latex")
xlabel("Transformed Feature",Interpreter=Style.Interpreter)
ylabel("y",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
nexttile
PlotResiduals(TransformAnalysis.BestModel,"Best")
disp("Best transformation by validation RMSE: " + TransformAnalysis.BestName)
%%
%[text] ## From Transformations to Rich Feature Sets
%[text] Thus far, each example has transformed a single predictor at a time. In many machine learning applications, however, feature engineering moves beyond one transformed predictor and into **feature construction**, where multiple engineered features are created simultaneously.
%[text] Rather than searching for one ideal transformation, we construct a richer feature set that captures different aspects of the underlying relationship. Some engineered features are single-predictor transformations, while others combine predictors or encode additional structure. Common engineered features include
%[text] - polynomial terms for nonlinear trends,
%[text] - trigonometric functions for periodic behavior,
%[text] - interaction terms that describe combined effects, and
%[text] - logarithmic or square-root transformations for scaling. \
%[text] The challenge is no longer finding one good transformation, but it is identifying the combination of engineered features that provides the best balance between predictive performance and generalization.
%[text]  **Exercise 3.** Is it correct to say that a feature transformation mainly improves an already engineered feature?
%[text] a. Yes. A transformation usually comes after feature engineering and improves an already engineered feature.
%[text] b. No. A transformed feature is itself an engineered feature, while construction is the broader step of assembling multiple engineered predictors.
%[text] c. Yes. Transformation is only useful after feature selection.
%[text] d. No. Transformation and construction are unrelated ideas.
CheckAnswer("ExerciseTrasnsformMiconception","Select") %[control:dropdown:51b5]{"position":[46,54]}
%%
%[text] ### Problem Setup: Observed Predictors, Hidden Response Structure
%[text] Feature construction is most useful when the raw predictors do not expose the full relationship between the inputs and the response. In this example, both $x${"altText":"x"} and $z${"altText":"z"} are observed predictors. The word *hidden* refers to the nonlinear structure in the response, not to a missing predictor.
%[text]  **Pro-tip**. To keep the visualization 2-D, the data is split into low, medium, and high bands of $z${"altText":"z"}. This makes it easier to see how the relationship between $x${"altText":"x"} and $y${"altText":"y"} changes as $z${"altText":"z"} changes.
%[text]  **Try**. Click **Generate Data with Hidden Structure** to generate the dataset and inspect the observed relationship.
%[control:button:9b34]{"position":[1,2]}
Traw = MakeConstructionDataset();
Style = DefaultPlotStyle();
PredictionSlices = MakePredictionSlices(Traw);
figure(Units="normalized",Position=[0.05 0.05 0.95 0.45])
PlotObservedSlices(Traw,PredictionSlices,"Observed Data by $z$ Band",Style)
%%
%[text] ### Baseline Model with Raw Features
%[text] The baseline model uses only the original predictors $x${"altText":"x"} and $z${"altText":"z"}. This gives you a reference point before any new predictors are added.
%[text]  **Try**. Click **Run Baseline Model (Raw Features)** to fit the raw-feature regression model and compare representative $z${"altText":"z"} slices of that model.
%[control:button:97db]{"position":[1,2]}
Traw = MakeConstructionDataset();
Style = DefaultPlotStyle();
PredictionSlices = MakePredictionSlices(Traw);
mdlRaw = fitlm(Traw,"y ~ x + z");
BaselineCurves = PredictModelSlices(mdlRaw,PredictionSlices);
disp(mdlRaw)
PredictionSets = struct( ...
"Label","Baseline: $y \sim x + z$", ...
"LineStyle","-", ...
"SeriesIndex",Style.FitSeriesIndex, ...
"YValues",{BaselineCurves});
figure(Units="normalized",Position=[0.05 0.05 0.95 0.45])
PlotSlicePredictions(Traw,PredictionSlices,PredictionSets, ...
"Baseline Model Across Representative $z$ Slices",Style)
%[text]  **Pro-tip**. Use these three baseline slices as the benchmark for the engineered models that follow.
%%
%[text] ### Create Engineered Features
%[text] Feature construction expands the design matrix with new columns that encode patterns the raw inputs do not express directly. In this example, some new columns are transformed versions of one predictor, such as $x^2${"altText":"x^2"}, $x^3${"altText":"x^3"}, $\\sin(x)${"altText":"sin(x)"}, $\\cos(x)${"altText":"cos(x)"}, $\\log(x)${"altText":"log(x)"}, and $\\sqrt{x}${"altText":"sqrt(x)"}. The interaction term $x\_z${"altText":"x\_z"} is different: it is a constructed combination feature that uses both original predictors together.
%[text]  **Try**. Click **Create Engineered Features** to create the engineered feature table and inspect the new predictors.
%[control:button:7428]{"position":[1,2]}
Traw = MakeConstructionDataset();
T = AddEngineeredFeatures(Traw);
disp("First rows of the engineered dataset:")
disp(head(T))
disp("The new columns create alternative representations of the original predictors without changing the regression algorithm.")
%%
%[text] ### Compare Candidate Feature Sets
%[text] Instead of guessing the best engineered model by eye, compare several candidate feature sets using a train-validation split. Training error measures how well the model fits what it has seen; validation error measures whether that improvement generalizes.
%[text]  **Try**. Click **Compare Feature Sets** to compare the candidate feature sets and rank them by validation MSE.
%[control:button:76b7]{"position":[1,2]}
Analysis = AnalyzeConstructionModels();
Style = DefaultPlotStyle();
figure
b = bar([Analysis.TrainMSE Analysis.ValMSE]);
b(1).SeriesIndex = Style.DataSeriesIndex;
b(2).SeriesIndex = Style.FitSeriesIndex;
xticklabels(string({Analysis.FeatureSets.Name}))
xtickangle(20)
ylabel("MSE",Interpreter=Style.Interpreter)
title("Feature Set Comparison: Training Versus Validation Error",Interpreter=Style.Interpreter)
legend("Train MSE","Validation MSE",Location="best",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style)
disp("Validation ranking from best to worst:")
disp(Analysis.Results)
%%
%[text] ### Visualize the Best Model
%[text] After identifying the best-performing feature set numerically, compare its predictions across the same representative slices used in the baseline plot. The best model combines low validation error with predictions that closely follow the observed pattern in the data.
%[text]  **Try**. Click **Visualize Best Model** to visualize the best-performing engineered model.
%[control:button:7985]{"position":[1,2]}
Analysis = AnalyzeConstructionModels();
Style = DefaultPlotStyle();
BestCurves = PredictModelSlices(Analysis.BestModel,Analysis.PredictionSlices);
PredictionSets = struct( ...
"Label","Best model", ...
"LineStyle","-", ...
"SeriesIndex",Style.FitSeriesIndex, ...
"YValues",{BestCurves});
figure(Units="normalized",Position=[0.05 0.05 0.95 0.45])
PlotSlicePredictions(Analysis.Traw,Analysis.PredictionSlices,PredictionSets, ...
"Best Feature Set by Validation: " + Analysis.BestName,Style)
disp("Best feature set: " + Analysis.BestName)
disp("Best model formula: " + Analysis.BestFormula)
%%
%[text] ## From Feature Engineering to Feature Selection
%[text] The previous examples compared several predefined feature sets. In practice, however, feature engineering often produces many candidate predictors simultaneously. A model may contain polynomial terms, interaction terms, trigonometric functions, and other transformations all at once.
%[text] While a richer feature set can improve predictive performance, including every engineered feature is rarely the best strategy. Some predictors contribute little information, while others may be highly correlated or increase the risk of overfitting.
%[text] The next step is **feature selection** which is identifying the subset of engineered features that provides the best balance between model complexity and predictive performance. One classical approach is **stepwise regression**, which automatically adds or removes predictors according to a statistical criterion.
%%
%[text] ### Feature Selection with Stepwise Regression
%[text] Stepwise regression searches a candidate pool of engineered predictors by repeatedly adding or removing terms according to a selection criterion. Here, the candidate pool is the full set of engineered features. In this particular example, we begin with the following model:
%[text]{"align":"center"} $y = x + z + x^2 + x^3 + sinx + \\cos(x) + x\_z + \\log(x) + \\sqrt{x}${"altText":"y = x + z + x^2 + x^3 + sinx + \cos(x) + x\_z + \log(x) + \sqrt{x}"}
%[text] Stepwise regression can be useful for exploring candidate predictors, but it is a greedy search procedure. Because it repeatedly evaluates many models, it can be unstable and may select different features when the data changes slightly.
%[text] For this reason, modern workflows often combine domain knowledge, validation data, and regularization methods such as Lasso rather than relying exclusively on stepwise selection.
%[text]  **Try**. Click **Run Stepwise Model** to evaluate the selected model on the validation split.
%[control:button:81a1]{"position":[1,2]}
Analysis = AnalyzeConstructionModels();
CandidateFormula = "y ~ x + z + x2 + x3 + sinx + cosx + x_z + logx + sqrtx";
mdlStep = stepwiselm(Analysis.Ttr,"constant","Upper",CandidateFormula,"Criterion","aic","Verbose",0);
yhatTrStep = predict(mdlStep,Analysis.Ttr);
yhatVaStep = predict(mdlStep,Analysis.Tva);
mseTrStep = mean((Analysis.Ttr.y - yhatTrStep).^2);
mseVaStep = mean((Analysis.Tva.y - yhatVaStep).^2);
disp(mdlStep)
disp("Selected stepwise model formula:")
disp(mdlStep.Formula)
disp("Stepwise regression performance:")
disp(table(["Train"; "Validation"],[mseTrStep; mseVaStep],VariableNames=["Split","MSE"]))
%%
%[text] ## From Feature Selection to Regularization
%[text] Stepwise regression reduces model complexity by selecting a subset of engineered features. However, it does so through a sequence of discrete include-or-exclude decisions, which can make the final model sensitive to changes in the training data, particularly when predictors are highly correlated.
%[text] Regularization takes a different approach. Instead of removing predictors, it modifies the optimization objective by adding a penalty term that discourages large regression coefficients. This penalty reduces the influence of less important predictors and helps improve generalization to new data.
%[text] If we start with the linear regression optimization objective ( [$\\star${"altText":"\star"}](file:./OptimizationandGradientDescentforRegression.m:M_29e7)),
%[text]{"align":"center"} $J(\\theta\_1,\\theta\_0) = \\frac{1}{n} \\sum\_{i=1}^{n} (\\theta\_1x\_i+\\theta\_0-y\_i)^2${"altText":"J(\theta\_1,\theta\_0) = \frac{1}{n} \sum\_{i=1}^{n} (\theta\_1x\_i+\theta\_0-y\_i)^2"} ,
%[text] regularization adds a penalty term controlled by the regularization parameter, $\\lambda${"altText":"\lambda"} :
%[text]{"align":"center"} **Regularization Expression Generalized**: $J(\\theta) = \\frac{1}{n} \\sum\_{i=1}^{n} (\\hat y\_i-y\_i)^2 + \\lambda \\cdot \\text{Penalty}${"altText":"J(\theta) = \frac{1}{n} \sum\_{i=1}^{n} (\hat y\_i-y\_i)^2 + \lambda \cdot \text{Penalty}"}
%[text] As $\\lambda${"altText":"\lambda"} increases, the penalty becomes stronger, encouraging smaller coefficient values and reducing model complexity.
%[text] Ridge regression uses an L2 penalty based on the squared coefficient values:
%[text]{"align":"center"} **Ridge Regression**: $J(\\theta) = \\frac{1}{n} \\sum\_{i=1}^{n} (\\hat y\_i-y\_i)^2 + \\lambda \\sum\_{j=1}^{p} \\theta\_j^2${"altText":"J(\theta) = \frac{1}{n} \sum\_{i=1}^{n} (\hat y\_i-y\_i)^2 + \lambda \sum\_{j=1}^{p} \theta\_j^2"},
%[text] while Lasso regression uses an L1 penalty based on the absolute coefficient values:
%[text]{"align":"center"} **Lasso Regression**: $J(\\theta) = \\frac{1}{n} \\sum\_{i=1}^{n} (\\hat y\_i-y\_i)^2 + \\lambda \\sum\_{j=1}^{p} |\\theta\_j|${"altText":"J(\theta) = \frac{1}{n} \sum\_{i=1}^{n} (\hat y\_i-y\_i)^2 + \lambda \sum\_{j=1}^{p} |\theta\_j|"}
%[text] Although both methods shrink coefficients, they do so differently. Ridge regression reduces the magnitude of all coefficients while retaining every predictor in the model, asking "How much should each feature contribute?" Lasso regression can shrink some coefficients exactly to zero, effectively performing automatic feature selection by asking "Which features should stay and which can be removed?"
%%
%[text] ### Regularization with Lasso and Ridge Regression
%[text] As we compare the coefficient paths for Lasso and Ridge, notice the differences in which features are removed or kept. Each path tracks the coefficient for a single feature. As the regularization penalty increases, Lasso repeatedly asks, "Which features are worth keeping?" Features that add little predictive value have their coefficients driven all the way to zero and are effectively removed from the model. As the regularization penalty increases, Ridge repeatedly asks, "How much should each feature contribute?" Coefficients become smaller, but features are rarely removed completely.
%[text]  **Try**. Click **Show Paths** to compare how Lasso and Ridge change feature coefficients as regularization increases.
RegularizationAnalysis = AnalyzeRegularizationModels();
Style = DefaultPlotStyle();
%%\ Visualization: paths and final Models
% Lasso coefficient paths
%[control:button:6abb]{"position":[1,2]}
figure(Units="normalized",Position=[0.05 0.05 0.9 0.6]);
tiledlayout(1,2)
nexttile
semilogx(RegularizationAnalysis.Lasso.FitInfo.Lambda,RegularizationAnalysis.Lasso.Path',LineWidth=1.2)
xline(RegularizationAnalysis.Lasso.Lambda,"--","Chosen $\lambda$",...
LabelOrientation="horizontal",Interpreter=Style.Interpreter)
xlabel("Regularization strength ($\lambda$)",Interpreter=Style.Interpreter)
ylabel("Feature influence (coefficient)",Interpreter=Style.Interpreter)
title("Lasso: Features are removed as regularization increases",...
Fontweight="bold",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style)
% Ridge coefficient paths
nexttile
semilogx(RegularizationAnalysis.Lambdas,RegularizationAnalysis.Ridge.Path',LineWidth=1.2)
xline(RegularizationAnalysis.Ridge.Lambda,"--","Chosen $\lambda$",...
LabelOrientation="horizontal",Interpreter=Style.Interpreter)
xlabel("$\lambda$",Interpreter=Style.Interpreter)
ylabel("Standardized coefficient",Interpreter=Style.Interpreter)
title("Ridge: All features stay, but their influence shrinks",...
Fontweight="bold",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style)
legend(RegularizationAnalysis.FeatureNamesLatex,Interpreter=Style.Interpreter,Location="eastoutside")
%[text]  **Reflect**. Which coefficients does Lasso drive completely to zero? Which coefficients remain nonzero in the Ridge model, even for large values of $\\lambda${"altText":"\lambda"}?
%%
%[text] #### Compare Models
%[text]  **Try**. Click **Compare Models** to compare the selected penalty values, coefficient estimates, and prediction errors for the final Lasso and Ridge Models.
%[control:button:1dd9]{"position":[1,2]}
%%\ Results
RegularizationAnalysis = AnalyzeRegularizationModels();
disp("Final regularized model comparison:")
disp(RegularizationAnalysis.PerformanceSummary)
disp("Chosen penalty values:")
disp(table(["Lasso";"Ridge"],[RegularizationAnalysis.Lasso.Lambda;RegularizationAnalysis.Ridge.Lambda],VariableNames=["Model","Lambda"]))
disp("Final standardized coefficients:")
disp(RegularizationAnalysis.CoefficientSummary)
%[text]  **Reflect**. Which model achieved the lower validation MSE? Did removing features noticeably improve or reduce predictive performance?
%%
%[text] #### Compare Fits
%[text]  **Try**. Click **Compare Fits** to compare the final Lasso and Ridge fits across the same representative $z${"altText":"z"} slices.
%%\ Compare final fits
%[control:button:5eae]{"position":[1,2]}
RegularizationAnalysis = AnalyzeRegularizationModels();
Style = DefaultPlotStyle();
PredictionSets = [
struct("Label","Lasso","LineStyle","-","SeriesIndex",Style.DataSeriesIndex,...
"YValues",{RegularizationAnalysis.Lasso.YPredictionSlices})
struct("Label","Ridge","LineStyle","--","SeriesIndex",Style.FitSeriesIndex,...
"YValues",{RegularizationAnalysis.Ridge.YPredictionSlices})
];
figure(Units="normalized",Position=[0.05 0.05 0.95 0.45])
PlotSlicePredictions( ...
RegularizationAnalysis.ConstructionAnalysis.Ttr, ...
RegularizationAnalysis.ConstructionAnalysis.PredictionSlices, ...
PredictionSets, ...
"Final Fits from Lasso and Ridge",Style)
%%
%[text] ## Summary
%[text] In this lesson, we explored how feature engineering can improve regression models by changing how predictors are represented. We compared feature transformations using cross-validation, constructed richer feature sets using polynomial, trigonometric, and interaction terms, and evaluated their impact on model performance. Finally, you examined feature selection and regularization techniques, including stepwise regression, Lasso, and Ridge, to balance model flexibility with generalization.
%[text] **Key Takeaways**
%[text] - Feature engineering is the umbrella process; feature transformation and feature construction are two different ways to engineer predictors.
%[text] - Feature transformations can reveal relationships that are difficult to model using raw predictors alone by re-expressing one predictor.
%[text] - Feature construction builds a broader candidate feature set that can include transformed predictors and interaction terms.
%[text] - Cross-validation helps identify Models that generalize well to new data.
%[text] - Engineered features can improve predictive performance, but excessive complexity can lead to overfitting, so feature selection and regularization help control that risk.
%[text] - Lasso and Ridge regularization help control model complexity and improve generalization. \
%[text] ## Further Exploration
%[text] Review the [Additional Resources](file:../InstructorResources/AdditionalResources.m:M_0e5b) section to find more materials for further learning.
%%
%[text] [⇦ Return to Main Menu](file:../MainMenu.m)
%%
%[text] ## Help us Improve
%[text] [Share your feedback](<matlab: ShareFeedback>) to help us improve this module.
%%
%[text] ## Local Helper Functions
%[text] If you wish to see the implementation details, switch to **Output Inline** from the **View** tab.
function [x,y,label] = MakeManualDataset(datasetChoice)
rng("shuffle")
x = rand(21,1);
switch datasetChoice
case "Periodic signal"
y = 0.5 + 0.5*sin(2*pi*x) + 0.21*rand(size(x));
label = "Periodic Signal";
case "Quadratic trend"
y = -1 + 2*x.^2 + 0.1*rand(size(x));
label = "Quadratic Trend";
case "Cubic trend"
y = 0.5 + 0.8*(x.^3 + 2*x.^2 + x) + 0.2*rand(size(x));
label = "Cubic Trend";
case "Mixed nonlinear signal"
y = 1 - 2*(x.^2 + sin(4*pi*x) + sin(2*pi*x)) + 0.2*rand(size(x));
label = "Mixed Nonlinear Signal";
otherwise
error("Unknown manual dataset selection.")
end
end
%%
function CheckAnswer(ExerciseID,Option)
arguments
ExerciseID (1,1) string
Option (1,1) string
end
Exercises = struct();
Exercises.ExerciseFeatureHierarchy = struct( ...
"Answer","a.", ...
"Hint","Feature engineering is the umbrella idea. Transformation and construction are two different ways to engineer features.");
Exercises.ExerciseTransformVsConstruct = struct( ...
"Answer","b.", ...
"Hint","Transformation re-expresses one predictor. Construction builds a broader candidate feature set that may include several transformed and combined predictors.");
Exercises.ExerciseTransformMisconception = struct( ...
"Answer","b.", ...
"Hint","A transformed feature is already an engineered feature. Construction is the broader step of assembling multiple engineered predictors.");
ExerciseName = char(ExerciseID);
if isfield(Exercises,ExerciseName)
CorrectAnswer = Exercises.(ExerciseName).Answer;
HintText = Exercises.(ExerciseName).Hint;
if Option == "Select"
return
elseif Option == CorrectAnswer
disp("You are correct.")
else
warning("%s",HintText)
end
end
end
%%
function [x,y,label] = MakeCVDataset(datasetChoice)
n = 100;
x = linspace(1,10,n)';
rng(0,"twister")
switch datasetChoice
case "Exponential growth"
y = exp(0.3*x) + randn(n,1);
label = "Exponential Growth";
case "Logarithmic growth"
y = 3*log(x) + 0.5*randn(n,1);
label = "Logarithmic Growth";
case "Sinusoidal pattern"
y = 0.8*sin(0.5*x) + 0.5*randn(n,1);
label = "Sinusoidal Pattern";
otherwise
error("Unknown cross-validation dataset selection.")
end
end
%%
function TransformAnalysis = EvaluateTransformLibrary(x,y)
BaseTransforms = [
struct("Name","x","Latex","$x$","Fcn",@(x) x)
struct("Name","x^2","Latex","$x^2$","Fcn",@(x) x.^2)
struct("Name","x^3","Latex","$x^3$","Fcn",@(x) x.^3)
struct("Name","log(x)","Latex","$\log(x)$","Fcn",@(x) SafeLog(x))
struct("Name","sqrt(x)","Latex","$\sqrt{x}$","Fcn",@(x) SafeSqrt(x))
struct("Name","1/x","Latex","$1/x$","Fcn",@(x) SafeReciprocal(x))
];
BaseTransforms = BaseTransforms(:).';
ExpScales = [0.1 0.3 0.5];
ExpTransforms = repmat(struct("Name","","Latex","","Fcn",[]),1,numel(ExpScales));
for i = 1:numel(ExpScales)
scale = ExpScales(i);
ExpTransforms(i) = struct( ...
"Name","exp(" + num2str(scale) + "*x)", ...
"Latex","$e^{" + num2str(scale) + "x}$", ...
"Fcn",@(x) exp(scale.*x));
end
Periods = GetPeriodLibrary();
PeriodicTransforms = repmat(struct("Name","","Latex","","Fcn",[]),1,2*numel(Periods));
for i = 1:numel(Periods)
idx = 2*i - 1;
period = Periods(i).Value;
label = string(Periods(i).Label);
latexLabel = string(Periods(i).LatexLabel);
PeriodicTransforms(idx) = struct( ...
"Name","sin(2*pi*x/" + label + ")", ...
"Latex","$\sin(2\pi x / " + latexLabel + ")$", ...
"Fcn",@(x) sin(2*pi*x./period));
PeriodicTransforms(idx + 1) = struct( ...
"Name","cos(2*pi*x/" + label + ")", ...
"Latex","$\cos(2\pi x / " + latexLabel + ")$", ...
"Fcn",@(x) cos(2*pi*x./period));
end
TransformLibrary = [BaseTransforms ExpTransforms PeriodicTransforms];
nT = numel(TransformLibrary);
Models = cell(nT,1);
Results = table(...
Size=[nT 6],...
VariableTypes=["string","double","double","double","double","logical"], ...
VariableNames=["Transform","MSE","MSECV","Rsq","Index","IsBest"]);
for i = 1:nT
TransformSpec = TransformLibrary(i);
xT = TransformSpec.Fcn(x);
valid = isfinite(xT) & isfinite(y);
xT = xT(valid);
yT = y(valid);
mdl = fitlm(xT,yT);
Models{i} = mdl;
Results.Transform(i) = TransformSpec.Name;
Results.MSE(i) = mdl.RMSE^2;
Results.MSECV(i) = ComputeCrossValidatedMSE(xT,yT,5);
Results.Rsq(i) = mdl.Rsquared.Ordinary;
Results.Index(i) = i;
end
Results = sortrows(Results,"MSECV");
Results.IsBest(:) = false;
Results.IsBest(1) = true;
BestTransform = TransformLibrary(Results.Index(1));
BestModel = Models{Results.Index(1)};
TransformAnalysis = struct();
TransformAnalysis.Transforms = TransformLibrary;
TransformAnalysis.Models = Models;
TransformAnalysis.Results = Results;
TransformAnalysis.BestModel = BestModel;
TransformAnalysis.BestFcn = BestTransform.Fcn;
TransformAnalysis.BestName = BestTransform.Name;
TransformAnalysis.BestLatex = BestTransform.Latex;
end
%%
function PlotResiduals(mdl,label)
arguments
mdl
label (1,1) string = ""
end
Style = DefaultPlotStyle();
yhat = mdl.Fitted;
res = mdl.Residuals.Raw;
scatter(yhat,res,Style.SmallMarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha)
hold on
yline(0,"k--",LineWidth=1.2)
hold off
xlabel("Fitted Values",Interpreter=Style.Interpreter)
ylabel("Residuals",Interpreter=Style.Interpreter)
title("Residuals vs Fitted: " + label,Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
end
%%
function mse = ComputeCrossValidatedMSE(x,y,K)
arguments
x (:,1) double
y (:,1) double
K (1,1) double {mustBePositive,mustBeInteger}
end
rng(11,"twister")
cv = cvpartition(numel(y),"KFold",K);
mseFold = zeros(K,1);
for k = 1:K
idxTrain = training(cv,k);
idxTest = test(cv,k);
mdl = fitlm(x(idxTrain),y(idxTrain));
yPred = predict(mdl,x(idxTest));
mseFold(k) = mean((y(idxTest) - yPred).^2);
end
mse = mean(mseFold);
end
%%
function [mdlRaw,mdlFeat,summary] = TestFeature(x,y,Feature,opts)
%TESTFEATURE Compare regression using x versus a transformed feature.
arguments
x (:,1) double
y (:,1) double
Feature (1,1) struct
opts.LabelInterpreter (1,1) string {mustBeMember(opts.LabelInterpreter,["none","tex","latex"])} = "latex"
opts.Alpha (1,1) double {mustBeGreaterThan(opts.Alpha,0),mustBeLessThan(opts.Alpha,1)} = 0.05
opts.IntervalType (1,1) string {mustBeMember(opts.IntervalType,["curve","observation"])} = "curve"
opts.MarkerSize (1,1) double {mustBePositive} = 36
opts.PlotRaw (1,1) logical = true
opts.PlotFeature (1,1) logical = true
opts.ShowStats (1,1) logical = true
end
Style = DefaultPlotStyle();
%%\ Compute feature
feature = Feature.Fcn(x);
feature = double(feature(:));
%%\ Remove invalid values
mask = isfinite(x) & isfinite(y) & isfinite(feature);
x = x(mask);
y = y(mask);
feature = feature(mask);
%%\ Fit models
mdlRaw = fitlm(x,y);
mdlFeat = fitlm(feature,y);
%%\ Plot raw data
if opts.PlotRaw
figure
scatter(x,y,opts.MarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,SeriesIndex=Style.DataSeriesIndex)
xlabel("$x$",Interpreter=Style.Interpreter)
ylabel("$y$",Interpreter=Style.Interpreter)
title("Raw Data",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
end
%%\ Plot transformed feature
if opts.PlotFeature
figure
tiledlayout(1,2,TileSpacing="compact",Padding="compact")
%%\ Overlay
nexttile
scatter(x,y,opts.MarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,...
DisplayName="Data",SeriesIndex=Style.DataSeriesIndex)
hold on
featureScaled = rescale(feature,min(y),max(y));
[xSorted,idx] = sort(x);
featureInterp = interp1(xSorted,featureScaled(idx),linspace(min(x),max(x),300),"pchip");
plot(linspace(min(x),max(x),300),featureInterp,LineWidth=2,...
DisplayName=Feature.DisplayName,SeriesIndex=Style.FitSeriesIndex)
hold off
xlabel("$x$",Interpreter=Style.Interpreter)
ylabel("$y$",Interpreter=Style.Interpreter)
title("Feature Overlay",Interpreter=Style.Interpreter)
legend(Location="best",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
%%\ Feature space
nexttile
scatter(feature,y,opts.MarkerSize,"filled",...
MarkerFaceAlpha=Style.DataMarkerFaceAlpha,DisplayName="Data",...
SeriesIndex=Style.DataSeriesIndex)
hold on
FeatureSorted = sort(feature);
[yFit,yCI] = predict(mdlFeat,FeatureSorted,...
Alpha=opts.Alpha,...
Prediction=opts.IntervalType);
fill( ...
[FeatureSorted; flipud(FeatureSorted)],[yCI(:,1); flipud(yCI(:,2))],...
Style.ConfidenceFillColor,...
EdgeColor="none",...
FaceAlpha=0.4,...
DisplayName="Confidence Interval")
plot(FeatureSorted,yFit,LineWidth=2,DisplayName="Linear Fit",...
SeriesIndex=Style.FitSeriesIndex)
hold off
xlabel(Feature.Latex,Interpreter=opts.LabelInterpreter)
ylabel("$y$",Interpreter=Style.Interpreter)
title( ...
"Linear Regression Using " + Feature.DisplayName,Interpreter=Style.Interpreter)
legend(Location="best",Interpreter=Style.Interpreter)
ApplyDefaultAxesStyle(Style,AxisMode="padded")
end
%%\ Summary table
summary = table( ...
["Raw (y ~ x)";
"Transformed (y ~ " + Feature.SummaryName + ")"], ...
[mdlRaw.Rsquared.Ordinary;
mdlFeat.Rsquared.Ordinary], ...
[mdlRaw.Rsquared.Adjusted;
mdlFeat.Rsquared.Adjusted], ...
[mdlRaw.RMSE;
mdlFeat.RMSE], ...
VariableNames=["Model","R2","AdjR2","RMSE"]);
if opts.ShowStats
disp("Model Comparison")
disp(summary)
end
end
%%
function Analysis = AnalyzeConstructionModels()
Traw = MakeConstructionDataset();
T = AddEngineeredFeatures(Traw);
PredictionSlices = MakePredictionSlices(Traw);
rng(19,"twister")
cv = cvpartition(height(T),"HoldOut",0.3);
idxTr = training(cv);
idxVa = test(cv);
Ttr = T(idxTr,:);
Tva = T(idxVa,:);
FeatureSets = [
struct("Name","Raw: x + z","Formula","y ~ x + z")
struct("Name","Polynomial: x + z + x^2 + x^3","Formula","y ~ x + z + x2 + x3")
struct("Name","Trigonometric: x + z + sinx + cosx","Formula","y ~ x + z + sinx + cosx")
struct("Name","Interaction: x + z + xz","Formula","y ~ x + z + x_z")
struct("Name","Mixed: poly + trig + interaction","Formula","y ~ x + z + x2 + x3 + sinx + cosx + x_z")
];
nSets = numel(FeatureSets);
TrainMSE = zeros(nSets,1);
ValMSE = zeros(nSets,1);
Models = cell(nSets,1);
for i = 1:nSets
Models{i} = fitlm(Ttr,FeatureSets(i).Formula);
yhatTr = predict(Models{i},Ttr);
yhatVa = predict(Models{i},Tva);
TrainMSE(i) = mean((Ttr.y - yhatTr).^2);
ValMSE(i) = mean((Tva.y - yhatVa).^2);
end
[~,BestIdx] = min(ValMSE);
Results = table(string({FeatureSets.Name})',TrainMSE,ValMSE,VariableNames=["FeatureSet","TrainMSE","ValidationMSE"]);
Results = sortrows(Results,"ValidationMSE");
Results.Rank = (1:height(Results))';
Results = movevars(Results,"Rank","Before","FeatureSet");
Analysis = struct;
Analysis.Traw = Traw;
Analysis.T = T;
Analysis.Ttr = Ttr;
Analysis.Tva = Tva;
Analysis.PredictionSlices = PredictionSlices;
Analysis.FeatureSets = FeatureSets;
Analysis.Models = Models;
Analysis.TrainMSE = TrainMSE;
Analysis.ValMSE = ValMSE;
Analysis.Results = Results;
Analysis.BestModel = Models{BestIdx};
Analysis.BestName = string(FeatureSets(BestIdx).Name);
Analysis.BestFormula = string(FeatureSets(BestIdx).Formula);
end
%%
function RegularizationAnalysis = AnalyzeRegularizationModels()
ConstructionAnalysis = AnalyzeConstructionModels();
FeatureNames = ["x","z","x2","x3","sinx","cosx","x_z","logx","sqrtx"];
FeatureNamesLatex = ["$x$","$z$","$x^2$","$x^3$","$\sin(x)$","$\cos(x)$","$xz$","$\log(x)$","$\sqrt{x}$"];
PredictionSlices = ConstructionAnalysis.PredictionSlices;
XTrain = ConstructionAnalysis.Ttr{:,cellstr(FeatureNames)};
XValidation = ConstructionAnalysis.Tva{:,cellstr(FeatureNames)};
YTrain = ConstructionAnalysis.Ttr.y;
YValidation = ConstructionAnalysis.Tva.y;
[XTrainStandardized,MuX,SigmaX] = zscore(XTrain);
SigmaX(SigmaX == 0) = 1;
XValidationStandardized = (XValidation - MuX) ./ SigmaX;
Lambdas = logspace(-4,3,60);
rng(29,"twister")
[LassoPath,FitInfo] = lasso(XTrainStandardized,YTrain,"Lambda",Lambdas,"CV",5,"Standardize",false);
LassoIndex = FitInfo.IndexMinMSE;
Lasso = struct();
Lasso.Path = LassoPath;
Lasso.FitInfo = FitInfo;
Lasso.Index = LassoIndex;
Lasso.Lambda = FitInfo.Lambda(LassoIndex);
Lasso.Beta = LassoPath(:,LassoIndex);
Lasso.Intercept = FitInfo.Intercept(LassoIndex);
Lasso.YHatTrain = XTrainStandardized*Lasso.Beta + Lasso.Intercept;
Lasso.YHatValidation = XValidationStandardized*Lasso.Beta + Lasso.Intercept;
Lasso.TrainMSE = mean((YTrain - Lasso.YHatTrain).^2);
Lasso.ValidationMSE = mean((YValidation - Lasso.YHatValidation).^2);
Lasso.SelectedMask = Lasso.Beta ~= 0;
RidgePath = zeros(numel(FeatureNames),numel(Lambdas));
RidgeTrainMSE = zeros(numel(Lambdas),1);
RidgeValidationMSE = zeros(numel(Lambdas),1);
RidgeModels = cell(numel(Lambdas),1);
for i = 1:numel(Lambdas)
RidgeModels{i} = fitrlinear(XTrainStandardized,YTrain,Learner="leastsquares",Regularization="ridge",Lambda=Lambdas(i),Solver="lbfgs");
RidgePath(:,i) = RidgeModels{i}.Beta;
yhatTrain = predict(RidgeModels{i},XTrainStandardized);
yhatValidation = predict(RidgeModels{i},XValidationStandardized);
RidgeTrainMSE(i) = mean((YTrain - yhatTrain).^2);
RidgeValidationMSE(i) = mean((YValidation - yhatValidation).^2);
end
[~,RidgeIndex] = min(RidgeValidationMSE);
Ridge = struct();
Ridge.Path = RidgePath;
Ridge.Models = RidgeModels;
Ridge.PathTrainMSE = RidgeTrainMSE;
Ridge.PathValidationMSE = RidgeValidationMSE;
Ridge.Index = RidgeIndex;
Ridge.Lambda = Lambdas(RidgeIndex);
Ridge.Model = RidgeModels{RidgeIndex};
Ridge.Beta = Ridge.Model.Beta;
Ridge.Intercept = Ridge.Model.Bias;
Ridge.YHatTrain = predict(Ridge.Model,XTrainStandardized);
Ridge.YHatValidation = predict(Ridge.Model,XValidationStandardized);
Ridge.TrainMSE = mean((YTrain - Ridge.YHatTrain).^2);
Ridge.ValidationMSE = mean((YValidation - Ridge.YHatValidation).^2);
Lasso.YPredictionSlices = cell(numel(PredictionSlices),1);
Ridge.YPredictionSlices = cell(numel(PredictionSlices),1);
for k = 1:numel(PredictionSlices)
XSlice = PredictionSlices(k).Table{:,cellstr(FeatureNames)};
XSliceStandardized = (XSlice - MuX) ./ SigmaX;
Lasso.YPredictionSlices{k} = XSliceStandardized*Lasso.Beta + Lasso.Intercept;
Ridge.YPredictionSlices{k} = predict(Ridge.Model,XSliceStandardized);
end
CoefficientSummary = table(FeatureNames',Lasso.Beta,Ridge.Beta,VariableNames=["Feature","Lasso_StdCoef","Ridge_StdCoef"]);
PerformanceSummary = table(["Lasso";"Lasso";"Ridge";"Ridge"],["Train";"Validation";"Train";"Validation"],[Lasso.TrainMSE;Lasso.ValidationMSE;Ridge.TrainMSE;Ridge.ValidationMSE],VariableNames=["Model","Split","MSE"]);
RegularizationAnalysis = struct();
RegularizationAnalysis.ConstructionAnalysis = ConstructionAnalysis;
RegularizationAnalysis.FeatureNames = FeatureNames;
RegularizationAnalysis.FeatureNamesLatex = FeatureNamesLatex;
RegularizationAnalysis.XTrainStandardized = XTrainStandardized;
RegularizationAnalysis.XValidationStandardized = XValidationStandardized;
RegularizationAnalysis.MuX = MuX;
RegularizationAnalysis.SigmaX = SigmaX;
RegularizationAnalysis.Lambdas = Lambdas;
RegularizationAnalysis.Lasso = Lasso;
RegularizationAnalysis.Ridge = Ridge;
RegularizationAnalysis.CoefficientSummary = CoefficientSummary;
RegularizationAnalysis.PerformanceSummary = PerformanceSummary;
end
%%
function Traw = MakeConstructionDataset()
rng(7,"twister")
n = 140;
x = linspace(0,10,n)';
z = 0.4*x + randn(n,1);
y = 2 + 0.6*x + 1.7*sin(x) + 0.25*(x.*z) + 0.6*randn(n,1);
Traw = table(x,z,y);
end
%%
function T = AddEngineeredFeatures(T)
T.x2 = T.x.^2;
T.x3 = T.x.^3;
T.sinx = sin(T.x);
T.cosx = cos(T.x);
T.x_z = T.x .* T.z;
T.logx = log(T.x + 1e-6);
T.sqrtx = sqrt(T.x);
end
%%
function PredictionSlices = MakePredictionSlices(Traw)
SliceLabels = ["Low z band","Middle z band","High z band"];
zEdges = quantile(Traw.z,[0 1/3 2/3 1]);
PredictionSlices(1,3) = struct( ...
"Label","",...
"ZValue",NaN,...
"ZRange",zeros(1,2),...
"XGrid",zeros(0,1),...
"Table",table());
for k = 1:3
if k < 3
BandMask = Traw.z >= zEdges(k) & Traw.z < zEdges(k+1);
else
BandMask = Traw.z >= zEdges(k) & Traw.z <= zEdges(k+1);
end
ZBand = Traw.z(BandMask);
XBand = Traw.x(BandMask);
ZValue = median(ZBand);
XGrid = linspace(min(XBand),max(XBand),200)';
SliceTable = table(XGrid,ZValue*ones(size(XGrid)),zeros(size(XGrid)),...
VariableNames=["x","z","y"]);
SliceTable = AddEngineeredFeatures(SliceTable);
PredictionSlices(k) = struct( ...
"Label",SliceLabels(k), ...
"ZValue",ZValue, ...
"ZRange",[min(ZBand) max(ZBand)], ...
"XGrid",XGrid, ...
"Table",SliceTable);
end
end
%%