yennik16/2026-24679-extrudability-autogluon-classifier

Extrudability classifier (AutoGluon, tabular)

This model serves to classify parts as extrudable or not (Given a set of measurements and observed features of an object, binarily classify it as being able to be modeled using only extrusions or not).

Headline result, stated honestly: 66.7% (4/6) on the held-out test set, against a 50% majority baseline, with a 95% confidence interval running from 22.3% to 95.7%. Six objects cannot distinguish a useful model from a coin flip. The cross-validated figure over 23 objects — balanced accuracy 0.750 ± 0.250 — is the number worth quoting, and its standard deviation says most of what there is to say about how much this dataset can support.

Purpose

Given the measured bounding dimensions, wall thickness, hole and edge counts, silhouette family, and material of a part, predict extrudable (1) versus not_extrudable (0). This model is intended as an exercise to gain familiarity with creating models for CMU course 24679 HW2. It serves as an initial feasibility check as to whteher a part can easily be converted from sketch to stl, but is educational and not meant for actual industry use.

Data origin and splits

• Dataset: sunkaiwen/sketch2stl-parts-tabular, CC-BY-4.0. Not my own dataset and not my Project 1 partner's.
• 34 hand-measured desk-scale objects, split 70/15/15 over objects, stratified on the target, before augmentation; 400 label-preserving synthetic rows were then derived from the 23 training objects only.
• Splits were used exactly as shipped: train 423 rows (23 measured objects + synthetic descendants), validation 5 objects, test 6 objects. No re-splitting, because re-splitting would scatter jittered copies of held-out objects into training.
• Verified in the notebook: no object appears in two splits, no synthetic row descends from a held-out object, the holdout splits contain measured rows only, and every synthetic row's label matches both of its parents.

Features and target

Target: extrudable (0 = not extrudable, 1 = extrudable). Positive class = 1.

Numeric inputs (10): length_mm, width_mm, height_mm, wall_thickness_mm (all mm); n_through_holes, n_straight_edges (counts); aspect_ratio, flatness, hole_density (unitless); footprint_mm2 (mm²).

Categorical inputs (2): silhouette_class, material.

Excluded to prevent leakage:

• extrusion_depth_mm — the dataset's other shipped target. Audited in the notebook: it is greater than zero exactly when the label is 1, on 100.0% of rows, making it the label in disguise. Left in the feature matrix it would produce a perfect and meaningless model.
• object_name — flagged as label-revealing by the dataset card, and inherited by synthetic rows, so it also acts as an object identifier.

Excluded as provenance: object_id, source_object_id, split_source, aug_technique, split. These are used only for lineage checks and for grouping the cross-validation folds.

Preprocessing

Minimal and deliberate. Measurements keep their physical magnitudes in mm; no scaling, binning, or imputation is applied (the dataset has no missing values, which is asserted). Categorical levels are fixed across all three splits so that singleton levels such as wood and ceramic are encoded consistently. AutoGluon applies its own internal encoding on top of this.

Training setup

• AutoGluon TabularPredictor, autogluon.tabular==1.6.1, problem_type="binary", positive_class=1, eval_metric="balanced_accuracy".
• num_bag_folds=0, num_stack_levels=0, with the shipped validation objects passed explicitly as tuning_data. This is the key setting: ~95% of training rows are synthetic siblings of 23 parent objects, so AutoGluon's default internal re-splitting would place jittered copies of the same object on both sides of a fold boundary and report an optimistic internal score.
• Seed 24679 for all splits and shuffles under our control; AutoGluon seeds its own model internals separately and reports model_random_seed = 0.

Search: two passes

Search A — model portfolio. presets="medium_quality", time_limit=300 s. This preset fits a fixed portfolio of model families (LightGBM variants, XGBoost, CatBoost, random forests, extra trees, k-NN, neural networks) at their default hyperparameter configurations. It searches over families, not over hyperparameters, so any "hyperparameters" it reports for a winner are stock values.

Selected: WeightedEnsemble_L2, validation balanced accuracy 0.833. The appended WeightedEnsemble_L2 reduces to LightGBMXT with weight 1.0 — verified by identical predicted probabilities — so it is a passthrough, not a blend. Note also that AutoGluon fits the weighted ensemble's weights on the validation data and then scores it on that same data, so its leaderboard score is optimistically biased relative to every base model.

model	score_val	fit_time
XGBoost	0.833	3.369
NeuralNetTorch	0.833	3.347
LightGBMXT	0.833	2.314
WeightedEnsemble_L2	0.833	2.673
LightGBM	0.75	3.466
LightGBMLarge	0.75	2.789
RandomForestEntr	0.667	1.652
RandomForestGini	0.667	1.693
NeuralNetFastAI	0.5	1.439
ExtraTreesEntr	0.417	1.83
ExtraTreesGini	0.417	1.595

Four model families tie at exactly 0.833 on a 5-object validation set, where one object is worth 20 percentage points. That is a four-way tie broken by AutoGluon's internal ordering, not a finding about which family suits this problem.

Search B — hyperparameter optimization. Random search, num_trials=20, time_limit=300 s, over LightGBM:

hyperparameter	range
learning_rate	Real(0.01, 0.2), log scale
num_leaves	Int(4, 32)
min_data_in_leaf	Int(2, 20)
feature_fraction	Real(0.5, 1.0)

Ranges are deliberately narrow because 23 genuinely independent objects cannot support a deep tree.

Selected: WeightedEnsemble_L2, validation balanced accuracy 0.833.

model	score_val	fit_time
LightGBM/T9	0.833	0.841
LightGBM/T8	0.833	0.835
LightGBM/T5	0.833	0.844
LightGBM/T1	0.833	1.122
LightGBM/T13	0.833	1.251
WeightedEnsemble_L2	0.833	1.128
LightGBM/T15	0.75	0.818
LightGBM/T16	0.75	0.879
LightGBM/T14	0.75	0.961
LightGBM/T20	0.75	0.881
LightGBM/T7	0.75	0.9
LightGBM/T3	0.75	0.843
LightGBM/T19	0.75	0.837
LightGBM/T6	0.75	0.806
LightGBM/T4	0.75	0.911
LightGBM/T2	0.75	0.851
LightGBM/T17	0.75	0.914
LightGBM/T18	0.75	0.815
LightGBM/T10	0.75	0.863
LightGBM/T12	0.75	1.088
LightGBM/T11	0.75	1.037

Twenty trials produced exactly two distinct scores, 0.833 and 0.75. The search cannot rank its own trials, so what advanced to cross-validation was an arbitrary member of a five-way tie. That is a reportable finding about the evaluation set, not a failure of the search: at 5 validation objects, balanced accuracy can only take a handful of values.

Selected configuration

portfolio, model WeightedEnsemble_L2 (effectively LightGBMXT). Selected on the cross-validation mean below, not the 5-object validation score, under a rule fixed before the numbers were seen: prefer the tuned search only if it beats the portfolio's CV mean by more than 0.10; otherwise keep the simpler portfolio configuration.

Hyperparameters of the selected model — note these are AutoGluon's stock LightGBMXT values, not searched ones, which is exactly the distinction Search B was added to make visible:

• learning_rate: 0.05
• extra_trees: True
• seed: 0

Metrics

All metrics are unitless fractions in [0, 1].

Object-level 5-fold cross-validation over the 23 measured training objects. Folds are drawn over objects; all synthetic descendants of a held-out object — including rows whose second parent is held out — are removed from that fold's training data, and each fold is scored on held-out measured objects only. Both configurations were evaluated on identical object groups.

configuration	accuracy (mean)	accuracy (std)	balanced accuracy (mean)	balanced accuracy (std)
portfolio	0.830	0.172	0.750	0.250
tuned	0.740	0.082	0.717	0.126

Per fold:

fold	held-out objects	portfolio	tuned
1	5	0.500	0.750
2	5	1.000	0.750
3	5	0.750	0.500
4	4	0.500	0.833
5	4	1.000	0.750

Reading the portfolio-versus-tuned comparison

The gap between the two means is 0.033 — less than one held-out object across the whole 23-object pool. It does not establish that hyperparameter optimization hurt, and the selection rule's 0.10 margin exists to stop exactly that misreading.

The per-fold column is the more informative view. The tuned configuration wins folds 1 and 4 and loses 2, 3 and 5. Portfolio swings from chance (0.500) to perfect (1.000) and back; tuned stays inside a 0.500–0.833 band. Its standard deviation is half the portfolio's, 0.126 against 0.250. Portfolio's mean is carried entirely by its two perfect folds, and its two chance folds are the same coin landing the other way.

That is the constrained search space behaving as designed. Capping num_leaves at 32 and allowing min_data_in_leaf down to 2 with column subsampling produces conservative models that neither spike nor collapse. On a dataset this small, the higher floor is arguably the more useful property, and the portfolio configuration was kept for simplicity rather than because it is demonstrably better.

One caveat on the comparison's fairness: during cross-validation each fold re-runs the entire 20-trial search within a 90-second budget, roughly 4.5 seconds per trial, against the portfolio's 90 seconds for about 11 models. The tuned arm may not have completed its full search in every fold, so its numbers are a lower bound on what that configuration can do.

Selected configuration: balanced accuracy 0.750 ± 0.250 (mean ± s.d. across folds), accuracy 0.830. This is the headline figure. Because folds share objects indirectly through the augmentation lineage, read the standard deviation as a stability estimate rather than a textbook standard error. The CV mean also served as the selection criterion between two candidates, which makes it slightly optimistic.

Held-out test set (6 measured objects), used once:

	Accuracy	Balanced accuracy	Precision (extrudable)	Recall (extrudable)	F1 (extrudable)
Selected model (portfolio)	0.667	0.667	0.600	1.000	0.750
Training-majority baseline	0.500	0.500	0.500	1.000	0.667

Test accuracy 66.7% (4/6 objects), 95% Clopper–Pearson interval 22.3% to 95.7%. With 6 objects, one error moves accuracy by about 17 percentage points, so the interval — not the point estimate — is the honest summary. Note the error pattern: recall on extrudable is 1.000 while precision is 0.600, so both errors are non-extrudable objects called extrudable. A feasibility check that never rejects a printable part but sometimes approves an unprintable one is the more expensive direction of the two.

Limitations and ethical notes

1. Tiny, non-random source. 34 objects from one person's desk, over-representing stationery and small hardware. No furniture, textiles, or organic shapes. The model cannot be said to generalize to parts in general.
2. Single-annotator label. extrudable is a documented ±1 mm judgement made by one person with no independent check. A good score means the model recovered that person's rule, not that it learned extrusion physics.
3. Model selection is not identifiable at this sample size. Four families tie exactly on the 5-object validation set, and 20 hyperparameter trials produced two distinct scores. Treat the selected model as a reasonable choice, not as the best one.
4. Effective training size is ~23 objects, not 423 rows. The synthetic rows are echoes, so a high training score is expected and uninformative.
5. Thin categorical coverage. ceramic, cork, rubber, and silicone each appear about once, so predictions for those materials rest on almost no evidence.
6. Errors are asymmetric in the costly direction. Both test errors were false positives for extrudability.
7. The realistic harm is misplaced confidence. A model that looks accurate on six objects could be confidently wrong about a real user's part, wasting filament at best and producing a part that fails in use at worst. Any downstream use should treat the output as a hint a human checks, never as a gate.

No personal data is involved: the dataset contains measurements of inanimate objects only.

License

CC-BY-4.0, matching the source dataset. Attribution to Serena Sun for the dataset and to yennik16 for the model.

Hardware and compute budget

• Hardware: Google Colab CPU runtime (no GPU).
• Search A: 300 s. Search B: 300 s. Cross-validation: 2 configurations × 5 folds × 90 s.
• Total fitting budget: about 1500 s (25 minutes) of wall-clock compute, excluding installs and downloads.
• Versions: autogluon.tabular 1.6.1, seed 24679.

AI usage disclosure

This model was vibecoded in accordance with the CMU 24679 assignment requirements. I first fed claude the dataset and informed it that I planned to build a binary classifier for the extrudable target, asking it to confirm it would be a good fit with the project requirements (I ran into an issue on my first attempt with a different dataset due to lack of a clear target, so this was a sanity check before further development). I then went back and forth prompting it to identify the best model for this use case, but ultimately fed it the example notebook from class as well as it started to drift from what was done in lecture. I then prompted it to explain why autogluon was reported as best despite the tie, and ran a few checks to ensure that weighted ensemble was actually a blend and not a passthrough of a prior model. This revealed that it was indeed a passtrhough in the initial build and that my original hyperparameters were ensembler configuration knobs, not tuned hyperparameters. I then prompted it to remedy this, yielding the current setup with two distinct searches. Claude also generated initial comments and the datacard, which I then went back to verify/edit.

How to use

import zipfile, huggingface_hub
from autogluon.tabular import TabularPredictor

path = huggingface_hub.hf_hub_download(
    repo_id="yennik16/2026-24679-extrudability-autogluon-classifier",
    filename="autogluon_predictor_dir.zip",
    repo_type="model",
)
with zipfile.ZipFile(path) as zf:
    zf.extractall("predictor_dir")
predictor = TabularPredictor.load("predictor_dir")
predictor.predict(your_dataframe)
# columns: length_mm, width_mm, height_mm, wall_thickness_mm, n_through_holes,
#          n_straight_edges, aspect_ratio, flatness, footprint_mm2, hole_density,
#          silhouette_class, material

Reload with the same library versions used above; AutoGluon artifacts are not guaranteed portable across versions.