#1 in MLP Project – Code, Chaos, and a Bit of Madness

This is my first time writing and sharing something like this publicly. But after a wild ride in the Machine Learning Practice (MLP) Project at IIT Madras—where I managed to finish Rank #1 out of 1700+ students—I felt it was worth putting my work out there.

Kaggle Leaderboard

In this post, I’m sharing my complete journey—what worked, what bombed, what I learned, and how I tackled the competition from start to finish. If you're into machine learning, Kaggle-style comps, or just here for a good story—you’re in the right place.

Introduction

This project was part of the Machine Learning Practice (MLP) course at IIT Madras. It was designed as a full-fledged Kaggle-style competition where over 1700 students participated. The task? Predict whether a system is likely to get infected by malware, using telemetry data collected by antivirus software.

We were provided with three main files:

  • train.csv – containing labeled data
  • test.csv – for which predictions were to be made
  • sample_submission.csv – showing the required submission format

    • You can download the Datasets here: GitHub

    Importing Libraries

    I started with the usual suspects — numpy, pandas and some plotting friends like matplotlib and seaborn. Obviously, I didn’t import everything at once. Like any true coder, I added more only when the errors started shouting at me.

    Eventually, I ended up with a pretty packed import block that looked like I was preparing for war!

    jupyter_notebook.ipynb
    import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import chi2_contingency

pd.set_option('display.max_rows', 100)

from sklearn.preprocessing import MinMaxScaler, OrdinalEncoder, OneHotEncoder
from category_encoders import BinaryEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split, RandomizedSearchCV
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.naive_bayes import CategoricalNB
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.metrics import ConfusionMatrixDisplay, classification_report

    Initial Exploration

    Loading the Data

    The first step was simple — load the CSVs. No rocket science here, just the classic pandas.read_csv() and a prayer that the paths were correct on the first try.

    jupyter_notebook.ipynb
    train = pd.read_csv('/kaggle/input/System-Threat-Forecaster/train.csv')
test = pd.read_csv('/kaggle/input/System-Threat-Forecaster/test.csv')


train.head(5)

    Quick Overview

    After loading, I checked the structure of the dataset —

    • 100,000 rows in the training set
    • 76 Columns, including the target
    • 47 Numerical Features
    • 28 Categorical Features
    • 33 Features with missing values, but each has less than 1% missing, totally manageable!

    Honestly? Not too messy—more like a college student’s room than a full crime scene.

    jupyter_notebook.ipynb
    def initial_exploration(dataset):
    print(f"Shape of Dataset: {dataset.shape}")
    feature_matrix = dataset.drop(columns=['target'])
    label_vector = dataset['target']

    numerical_features = feature_matrix.select_dtypes(include='number').columns
    categorical_features = feature_matrix.select_dtypes(include='object').columns
    print(f"Numerical Features: {len(numerical_features)}")
    print(f"Categorical Features: {len(categorical_features)}")

    nan_features = dataset.columns[dataset.isna().sum() > 0]
    print(f"Features with NaN: {len(nan_features)}")
    return feature_matrix, label_vector, numerical_features, categorical_features

X_train, y_train, numerical_features, categorical_features = initial_exploration(train)
    jupyter_notebook.ipynb
    def null_values_analyze(dataset):
    custom_df = dataset.isna().sum()
    custom_df = custom_df[custom_df > 0].to_frame(name='NaN Count')
    custom_df['NaN %'] = (custom_df['NaN Count'] / dataset.shape[0]) * 100

    print(custom_df)

null_values_analyze(train)

    Exploratory Data Analysis

    Before diving into model building, I needed to understand the battlefield—aka, the dataset. Exploratory Data Analysis (EDA) helped identify patterns, suspicious features, potential data issues, and hidden gold mines.

    Numerical Features Overview

    Time to zoom into the numerical features. Some of them were helpful, some were just there for moral support.

    • Useless Columns: IsBetaUser, AutoSampleSubmissionEnabled, and IsFlightsDisabled had only a single unique value. Basically the interns of the dataset.
    • Binary Gang: Many columns like IsPassiveModeEnabled, IsSystemProtected, SMode, FirewallEnabled, HasOpticalDiskDrive, IsPortableOS, IsSecureBootEnabled, IsVirtualDevice, IsTouchEnabled, IsPenCapable, IsAlwaysOnAlwaysConnectedCapable, and IsGamer were binary—great for modeling, not so great for insight.
    • Genuinely Numeric: Features like ProcessorCoreCount, PrimaryDiskCapacityMB, SystemVolumeCapacityMB, TotalPhysicalRAMMB, PrimaryDisplayDiagonalInches, PrimaryDisplayResolutionHorizontal, and PrimaryDisplayResolutionVertical were actual continuous numerical variables. Finally, some real MVPs.
    • Important Distribution Patterns :
      1. TotalPhysicalRAMMB is right-skewed — most systems had lower RAM, with fewer high-end machines hogging the upper range.
      2. PrimaryDisplayResolutionHorizontal and PrimaryDisplayResolutionVertical clustered around common resolutions like 1366×768 and 1920×1080 — standard display gang.
      3. CountryID and CityID had broad, fairly uniform distributions, suggesting data came from a wide variety of locations—a nice bonus for generalization.
    • Interesting Correlations :
      1. PrimaryDisplayResolutionHorizontal and PrimaryDisplayResolutionVertical are 0.75 correlated, which is logical since screen dimensions often scale together.
      2. OSUILocaleID and OSInstallLanguageID have 0.99 correlation — twins, basically.
      3. OSBuildNumberOnly and OSBuildRevisionOnly show 0.95 correlation.
      4. TotalPhysicalRAMMB correlates with PrimaryDiskCapacityMB, indicating rich devices are rich in everything.
    jupyter_notebook.ipynb
    X_train[numerical_features].describe()

    Visualizing Outliers with Boxplots

    jupyter_notebook.ipynb
    plt.figure(figsize=(15, 40))

sns.boxplot(data=MinMaxScaler().fit_transform(train[numerical_features]), color="royalblue", orient="h", width=0.6)

plt.title("Box Plots of Numerical Features", fontweight="bold", fontsize=20)
plt.xlabel("Value", fontweight="bold", fontsize=12)
plt.ylabel("Numerical Variables", fontweight="bold", fontsize=20)
plt.yticks(ticks=list(range(len(numerical_features))), labels=numerical_features, fontsize=13)

plt.show()

    Distribution of Numerical Features

    jupyter-notebook.ipynb
    fig, axes = plt.subplots(16,3, figsize=(15,60))
axes = axes.flatten() 

for i, feature in enumerate(numerical_features):
    X_train[feature].plot(kind="hist", ax=axes[i], bins=12, color='royalblue', alpha=0.8)
    axes[i].set_title(f"{feature} Distribution", fontweight="bold")
    axes[i].set_ylabel("Frequency")

fig.tight_layout(pad=3.0)
plt.show()

    Correlation Heatmap for Numerical Features

    jupyter-notebook.ipynb
    plt.figure(figsize=(50, 40))
sns.heatmap(X_train[numerical_features].corr(), annot=True, cmap='coolwarm')
plt.show()

    Categorical Features Overview

    After taming the numerical features, it was time to dive into the land of categories — the drama queens of any dataset.

    • Dominant Classes :
      1. Platform Type: Windows 10 dominates the dataset (>95% of systems), with minimal presence of Windows 7, Windows 8, and Windows 2016.
      2. OS Architecture: AMD64 (64-bit) architecture is the majority (>90%), with minimal x86 (32-bit) and ARM64 systems.
      3. Device Family: Windows Desktop systems make up the vast majority, with very few Windows Server devices.
    • Redundancy Alert :
      1. Product Name, OS Version and Platform Type are tightly coupled — (99%) correlation. If you’ve seen one, you’ve seen them all.
      2. Processor and Architecture have perfect correlation (1.00). Makes sense—you’re not running 64-bit Windows on a potato CPU.

    Distribution of Categorical Features

    jupyter-notebook.ipynb
    filtered_features = [col for col in categorical_features if X_train[col].nunique() <= 10]

fig, axes = plt.subplots(len(filtered_features) // 2 + 1, 2, figsize=(14, len(filtered_features) * 1.5))
axes = axes.flatten()

for i, feature in enumerate(filtered_features):
    sns.countplot(y=X_train[feature], ax=axes[i], order=X_train[feature].value_counts().index, color="royalblue", alpha=0.8)
    axes[i].set_title(f"{feature} Distribution", fontweight="bold")
    axes[i].set_xlabel("Count")
    axes[i].set_ylabel("")

fig.tight_layout(pad=2.0)
plt.show()

    Correlation between Categorical Features (Cramér’s V Style)

    jupyter-notebook.ipynb
    def cramers_v(x, y):
    contingency_table = pd.crosstab(x, y)
    chi2, _, _, _ = chi2_contingency(contingency_table)
    n = contingency_table.sum().sum()
    r, k = contingency_table.shape
    return np.sqrt(chi2 / (n * (min(r, k) - 1)))

filtered_features = [col for col in categorical_features if X_train[col].nunique() <= 10]

categorical_corr = pd.DataFrame(index=filtered_features, columns=filtered_features, dtype=float)

for i, col1 in enumerate(filtered_features):
    for j, col2 in enumerate(filtered_features):
        if j >= i:
            if col1 == col2:
                categorical_corr.loc[col1, col2] = 1.0
            else:
                categorical_corr.loc[col1, col2] = cramers_v(X_train[col1], X_train[col2])
                categorical_corr.loc[col2, col1] = categorical_corr.loc[col1, col2]

plt.figure(figsize=(30, 15))
sns.heatmap(categorical_corr, annot=True, fmt=".2f", cmap='coolwarm', linewidths=0.5, square=True)
plt.title("Categorical Feature Correlation Heatmap (Cramér's V)", fontsize=14, fontweight="bold")
plt.xticks(rotation=90)
plt.yticks(rotation=0)
plt.show()

    Target Variable Analysis

    Before moving forward with modeling, I needed to verify if the target variable was balanced—or if I was about to walk into an imbalanced data nightmare.

    • The classes were almost perfectly balanced. Around 51% vs 49%.
    • No class dominates, which is a dream scenario in binary classification.
    • That means: No need for oversampling, undersampling, synthetic data generation, or praying to the ML gods.
    jupyter-notebook.ipynb
    labels = ['Class 1', 'Class 0']
sizes = y_train.value_counts()
colors = ['skyblue', 'salmon']
explode = (0.1, 0) 

plt.figure(figsize=(4,4))
plt.pie(sizes, labels=labels, autopct='%1.1f%%', colors=colors, explode=explode, shadow=True, startangle=140)
plt.title('Distribution of target')
plt.show()

    Data Preprocessing

    Alright, EDA gave me all the gossip: who’s fake, who’s flaky, and who’s actually useful. Then came the cleanup — like Mom before Diwali. No mercy, just dropping useless columns, engineering new ones, handling nulls, and encoding everything into a machine-learning-friendly format. Total sanskari data now.

    Removing Columns

    1. MachineID: just a random identifier, not useful for predictions.
    2. IsBetaUser, AutoSampleSubmissionEnabled, IsFlightsDisabled: Each had only one unique value. Why even show up?
    3. OSUILocaleID: 99% correlated with OSInstallLanguageID (BFFs)
    4. OSBuildNumberOnly: 95% correlated with OSBuildNumber (BFFs)
    5. Dropping additional features that are binary but have a single dominant value, as well as some features that are unlikely to be relevant.
    jupyter-notebook.ipynb
    drop_cols = ['AutoSampleSubmissionEnabled', 'IsBetaUser', 'IsFlightsDisabled', 
             'OSBuildNumberOnly', 'MachineID', 'ProductName', 'PlatformType', 'DeviceFamily',
             'OSUILocaleID', 'OSProductSuite', 'LocaleEnglishNameID', 'OSBuildRevisionOnly', 
             'OSBuildNumber', 'HasTpm', 'GeoRegionID', 'CountryID']

X_train_dropped = X_train.drop(columns=drop_cols)
test_dropped = test.drop(columns=drop_cols)

    Feature Engineering

    1. DateAS - DateOS: I had two dates: DateAS (Antivirus Date) and DateOS (OS Installation Date). On their own? Boring. But subtract one from the other? Boom — I got something meaningful, How long the antivirus came after the OS was installed? Turns out, Users who updated antivirus after the OS install were more likely to detect malware. Makes sense—defense came late.
    2. AppVersion: AppVersion was something like 4.18.2001.10 . Cool for humans, not great for models. I just grabbed the second part (18) to turn it into a clean integer.

    But hold up, this dataset also had other version like columns: SignatureVersion, NumericOSVersion, OSVersion, EngineVersion! These also looked like version numbers. So naturally, I tried. Trust me, I did. But here’s the twist...

    Breaking them down didn’t help the model much. Maybe because the version formats were inconsistent, or because too many micro-versions created noise. So instead of dissecting them, I passed them through an Ordinal Encoder!

    jupyter-notebook.ipynb
    # Feature Engineering 1
X_train_dropped['DiffOS'] = pd.to_datetime(X_train_dropped['DateAS']) - pd.to_datetime(X_train_dropped['DateOS'])
X_train_dropped['DiffOS'] = X_train_dropped['DiffOS'].map(lambda x: x.days//7)
test_dropped['DiffOS'] = pd.to_datetime(test_dropped['DateAS']) - pd.to_datetime(test_dropped['DateOS'])
test_dropped['DiffOS'] = test_dropped['DiffOS'].map(lambda x: x.days//7)

del X_train_dropped['DateAS'], X_train_dropped['DateOS'] 
del test_dropped['DateAS'], test_dropped['DateOS']

# Feature Engineering 2
X_train_dropped['AppVersion'] = X_train_dropped['AppVersion'].map(lambda x: int(x.split('.')[1]))
test_dropped['AppVersion'] = test_dropped['AppVersion'].map(lambda x: int(x.split('.')[1]))

    Handling Missing Values

    Since almost all features represent some kind of category (even if they are given as numerical), it's better to impute them with the most frequent value.

    jupyter-notebook.ipynb
    most_frequent_cols = ['SystemVolumeCapacityMB', 'InternalBatteryNumberOfCharges', 
    'RealTimeProtectionState', 'AntivirusConfigID', 'NumAntivirusProductsInstalled', 
    'NumAntivirusProductsEnabled', 'CityID', 'IEVersionID', 'ProcessorCoreCount',
    'EnableLUA', 'OEMNameID', 'OEMModelID', 'ProcessorManufacturerID', 'PrimaryDiskCapacityMB',
    'ProcessorModelID', 'OSInstallLanguageID', 'FirmwareManufacturerID', 'FirmwareVersionID', 
    'RegionIdentifier',  'PrimaryDisplayDiagonalInches', 'PrimaryDisplayResolutionHorizontal',
    'PrimaryDisplayResolutionVertical', 'DiffOS', 'TotalPhysicalRAMMB']

    Feature Encoding & Scaling

    Now that the data was trimmed, cleaned, and slightly smarter (thanks to feature engineering), it was time to translate everything into a format the models can actually understand—aka, numbers. Because sadly, scikit-learn doesn’t speak “WindowsDesktop.”

    • Binary Encoder: These were things like IsGamer, IsVirtualDevice etc. Pure 0 or 1 type features. For these, I used a Binary Encoder to avoid breaking the column into 2 separate ones like One-Hot Enocoder does. Cleaner and more memory-efficient.
    • OrdinalEncoder: Columns like OSVersion, Processor, OSEdition etc. had many unique values but still some kind of natural order (like newer versions being higher). For those, I used Ordinal Encoding. Why not OneHot? Because:
      • OneHot would explode the feature space (too many dummies).
      • These columns had thousands of categories in some cases.
      • Ordinal was faster, lighter, and honestly—performed better in testing.
    jupyter-notebook.ipynb
    binary_cols = ['IsPassiveModeEnabled', "IsSystemProtected", "SMode", "FirewallEnabled", 
               "HasOpticalDiskDrive", "IsPortableOS", "IsSecureBootEnabled", "IsVirtualDevice", 
               "IsTouchEnabled", "IsPenCapable", "IsAlwaysOnAlwaysConnectedCapable", "IsGamer"]


ordinal_cols = ["AppVersion", "SignatureVersion", "OSBuildLab", "MDC2FormFactor", "ChassisType", 
               "NumericOSVersion", "OSBranch", "OSEdition", "OSSkuFriendlyName", "FlightRing",
               "OSVersion", "Processor",  "OsPlatformSubRelease", "SKUEditionName", "PrimaryDiskType",
               "PowerPlatformRole", "OSArchitecture", "OSInstallType", "AutoUpdateOptionsName",
               "OSGenuineState", "LicenseActivationChannel", "EngineVersion"]

    The Preprocessing Pipeline

    Now I built a proper ColumnTransformer pipeline, so I could process everything in one go like a boss. This setup handled:

    • Imputation for missing values
    • Binary Encoding for binary features
    • Ordinal Encoding for multi-class categorical features
    • Passthrough for anything already numeric
    jupyter-notebook.ipynb
    column_transformer = ColumnTransformer(
    transformers=[
        ('most_frequent', SimpleImputer(strategy='most_frequent'), most_frequent_cols),

        ('binary', Pipeline([
            ('imputer', SimpleImputer(strategy='most_frequent')),
            ('binary', BinaryEncoder())
        ]), binary_cols),

        ('ordinal', Pipeline([
            ('imputer', SimpleImputer(strategy='most_frequent')),
            ('ordinal',  OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1))
        ]), ordinal_cols),

       
    ], remainder='passthrough')


column_transformer.fit(X_train_dropped)
X_train_transformed = column_transformer.transform(X_train_dropped)
test_transformed = column_transformer.transform(test_dropped)

    Scaling Features

    For scale-sensitive models like KNN, SVM, MLP, I scaled everything between 0 and 1 using MinMaxScaler. Not every model needs it, but for the ones that do? It’s the difference between “Haan decent” and “Okay, that’s actually smart...”

    jupyter-notebook.ipynb
    min_max_scaler = MinMaxScaler()

X_train_scaled = min_max_scaler.fit_transform(X_train_transformed)
test_scaled = min_max_scaler.transform(test_transformed)

    Model Building

    With the data now cleaned, encoded, and scaled, it was time to try out different classification models — starting from the absolute basics, and working up to the monsters like LightGBM and XGBoost.

    Train-Test Split

    Before training anything, I split the data to simulate a real-world scenario — training on 90% and validating on 10%. Why two splits? One for models that needed scaled data (like KNN, SVM), and one for models that didn’t (like tree-based models).

    jupyter-notebook.ipynb
    # For Scale-Sensitive Models
X_train_scaled, X_val_scaled, y_train_scaled, y_val_scaled = train_test_split(X_train_scaled, y_train, test_size=0.1, random_state=69)

# For Scale-Insensitive Models
X_train_transformed, X_val_transformed, y_train_transformed, y_val_transformed = train_test_split(X_train_transformed, y_train, test_size=0.1, random_state=69)

    Baseline Model

    I started with a DummyClassifier. Why? Because I needed to know the bare minimum a model had to beat to be taken seriously.

    Dummy model gave an accuracy of 0.511, which is expected for a binary classification problem.

    jupyter-notebook.ipynb
    dummy_classifier = DummyClassifier(random_state=69)
dummy_classifier.fit(X_train_scaled, y_train_scaled)

y_pred_dummy = dummy_classifier.predict(X_val_scaled)
dummy_accuracy = accuracy_score(y_val_scaled, y_pred_dummy)

print(f"Dummy Accuracy: {dummy_accuracy:.4f}")

    K-Nearest Neighbor

    Next, I brought in KNN — the classic “ask your neighbors” approach. Simple, intuitive, and surprisingly decent. KNN gave an accuracy of 0.5598, slightly better than the baseline (0.5110) and crosses the 0.55 cut-off. This indicates the model is learning patterns but is still weak.

    jupyter-notebook.ipynb
    knn_classifier = KNeighborsClassifier()
knn_classifier.fit(X_train_scaled, y_train_scaled)

y_pred_knn = knn_classifier.predict(X_val_scaled)
knn_accuracy = accuracy_score(y_val_scaled, y_pred_knn)

print(f"KNN Accuracy: {knn_accuracy:.4f}")

    Stochastic Gradient Descent

    SGD came in next — fast, efficient, but sometimes reckless. SGD gave an accuracy of 0.5946, showing improvement over KNN. The model is learning better patterns but still has room for optimization.

    jupyter-notebook.ipynb
    sgd_classifier = SGDClassifier(random_state=69)
sgd_classifier.fit(X_train_scaled, y_train_scaled)

y_pred_sgd = sgd_classifier.predict(X_val_scaled)
sgd_accuracy = accuracy_score(y_val_scaled, y_pred_sgd)

print(f"SGD Accuracy: {sgd_accuracy:.4f}")

    Logistic Regression

    Everyone’s first love in ML — Logistic Regression. Easy to explain, and surprisingly strong for linear problems. Logistic Regression achieved an accuracy of 0.5989, Slightly better than SGD, but not revolutionary.

    jupyter-notebook.ipynb
    logistic_regression = LogisticRegression(max_iter=500)
logistic_regression.fit(X_train_scaled, y_train_scaled)

y_pred_logistic = logistic_regression.predict(X_val_scaled)
logistic_accuracy = accuracy_score(y_val_scaled, y_pred_logistic)

print(f"Logistic Regression Accuracy: {logistic_accuracy:.4f}")

    Support Vector Machine

    SVM doesn’t mess around. It draws a boundary and dares your data to cross it. But... it's also slow as hell on large datasets. SVM hit an accuracy of 0.6013

    Finally crossing the 0.60 mark! Not bad, but training this felt like waiting for Indian trains during monsoon. While it crossed 0.60 but the gain is minimal—perhaps hyperparameter tuning can unlock more potential!

    jupyter-notebook.ipynb
    svm_classifier = SVC()
svm_classifier.fit(X_train_scaled, y_train_scaled)

y_pred_svm = svm_classifier.predict(X_val_scaled)
svm_accuracy = accuracy_score(y_val_scaled, y_pred_svm)

print(f"SVM Accuracy: {svm_accuracy:.4f}")

    Multi-Layer Perceptron

    Time to let deep learning dip its toes in. MLP is the classic go-to for dense, non-linear stuff. MLP achieved an accuracy of 0.6115, It outperforms SVM, showing deeper learning is paying off.

    jupyter-notebook.ipynb
    mlp_classifier = MLPClassifier(max_iter=500, random_state=1437)
mlp_classifier.fit(X_train_scaled, y_train_scaled)

y_pred_mlp = mlp_classifier.predict(X_val_scaled)
mlp_accuracy = accuracy_score(y_val_scaled, y_pred_mlp)

print(f"MLP Accuracy: {mlp_accuracy:.4f}")

    Random Forest

    Tree-based, fast, and powerful. It doesn’t care about scaling, outliers, or your excuses. Random Forest scored an accuracy of 0.6177, slightly surpassing MLP.

    Being a bagging-based model (Multiple Decision Trees trained in parallel on different subsets), it reduces variance. While it's performing well, the improvement is marginal—further tuning or feature selection might help boost it further!

    jupyter-notebook.ipynb
    rfc_classifier = RandomForestClassifier(random_state=69)
rfc_classifier.fit(X_train_transformed, y_train_transformed)

y_pred_rfc = rfc_classifier.predict(X_val_transformed)
rfc_accuracy = accuracy_score(y_val_transformed, y_pred_rfc)

print(f"Random Forest Accuracy: {rfc_accuracy:.4f}")

    Extreme Gradient Boosting

    As a boosting-based model, it sequentially improves weak learners, reducing both bias and variance for stronger predictions. XGBoost achieved an accuracy of 0.6251, marking the highest performance so far! It outperforms Random Forest, showing the power of boosting in capturing complex patterns.

    jupyter-notebook.ipynb
    xgb_classifier = XGBClassifier()
xgb_classifier.fit(X_train_transformed, y_train_transformed)

y_pred_xgb = xgb_classifier.predict(X_val_transformed)
xgb_accuracy = accuracy_score(y_val_transformed, y_pred_xgb)

print(f"XGBoost Accuracy: {xgb_accuracy:.4f}")

    LightGBM - The Silent Killer

    Faster than XGBoost. Lighter. Meaner. Cleaner. I saved this one for last. LGBM achieved an accuracy of 0.6292, taking the lead over XGBoost!

    Its efficiency and speed make it a strong contender for the best model. With some hyperparameter tuning, we might unlock even more potential!

    jupyter-notebook.ipynb
    lgbm_classifier = LGBMClassifier(verbose=-1)
lgbm_classifier.fit(X_train_transformed, y_train_transformed)

y_pred_lgbm = lgbm_classifier.predict(X_val_transformed)
lgbm_accuracy = accuracy_score(y_val_transformed, y_pred_lgbm)

print(f"LGBM Accuracy: {lgbm_accuracy:.4f}")

    Model Comparison

    And the winner for now? LightGBM. It delivered speed, accuracy, and didn’t eat all my RAM.

    Model Comparison

    jupyter-notebook.ipynb
    models = ["Dummy\nClassifier", "K-Nearest\nNeighbors", "Stochastic Gradient\nDescent",
          "Logistic\nRegression", "Support Vector\nMachine", "Multi-Layer\nPerceptron",
          "Random\nForest", "Extreme Gradient\nBoosting", "Light Gradient\nBoosting Machine"]

accuracies = [dummy_accuracy, knn_accuracy, sgd_accuracy, logistic_accuracy, svm_accuracy,
              mlp_accuracy, rfc_accuracy, xgb_accuracy, lgbm_accuracy]  


plt.figure(figsize=(20, 5))
sns.set_style("whitegrid")
colors = sns.color_palette("coolwarm", len(models))  

# Barplot
ax = sns.barplot(x=np.array(models), y=np.array(accuracies), palette=colors, edgecolor="black", width=0.6)

# Label
for bar, accuracy in zip(ax.patches, accuracies):
    ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 0.002, f"{accuracy:.3f}", 
            ha='center', va='bottom', fontsize=12, fontweight="bold", color="black")

plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
plt.ylim(0.50, 0.65) 
plt.xlabel("")
plt.ylabel("Accuracy", fontsize=14, fontweight="bold", labelpad=10)
plt.title("Model Accuracy Comparison", fontsize=16, fontweight="bold", pad=15)
sns.despine()
plt.show()

    Model Optimization

    Now here’s the thing — I’d been training and validating on 90% of the data. But since I was about to tune this model for submission...

    Now I trained the model on 100% of the data. Yes, it’ll overfit on this set. But who cares? I’m not evaluating anymore — I’m optimizing for the final submission. This stage is all about pushing the accuracy on the test set using the best-tuned parameters. Validation drama ends here.

    jupyter-notebook.ipynb
    X_train_transformed = column_transformer.transform(X_train_dropped)

    Tuning LightGBM

    I used RandomizedSearchCV to find the best combo of hyperparameters. Why Randomized and not GridSearch? Because I value my time, RAM, and sanity. I tuned three time and got accuracies of 0.7261, 0.7493 and 0.7284 respectively.

    Note: These are definitely overfitting slightly now since we’re using 100% of the data. But again, who cares? Now I just want to push accuracy on the private data, so obviously I’m using the full dataset.

    jupyter-notebook.ipynb
    param_dist = {
    'num_leaves': np.arange(100, 300, 7),  
    'max_depth': np.arange(5, 15, 1),
    'learning_rate': np.linspace(0.01, 0.1, 100),
    'n_estimators': np.arange(100, 300),
    'min_child_samples': np.arange(5, 30),
    'min_child_weight': np.logspace(-2, 1, 100),
    'subsample': np.linspace(0.7, 1.0, 100),
    'colsample_bytree': np.linspace(0.7, 1.0, 100),
    'reg_alpha': np.logspace(-3, 0, 100),
    'reg_lambda': np.logspace(-3, 0, 100)
}

lgbm = LGBMClassifier(verbose=-1)

random_search = RandomizedSearchCV(
    estimator=lgbm, 
    param_distributions=param_dist, 
    n_iter=50,
    scoring='accuracy',
    cv=5, 
    verbose=1, 
    n_jobs=-1,
    random_state=69
)

random_search.fit(X_train_transformed, y_train_transformed)
best_lgbm = random_search.best_estimator_
best_lgbm
    jupyter-notebook.ipynb
    # 3 LGBM Models
lgbm1 = LGBMClassifier(num_leaves=146, max_depth=11, learning_rate=0.04059333955483345, n_estimators=215, min_child_samples=12, min_child_weight=5.348307316066151, subsample=0.8015824978715238, colsample_bytree=0.7528665653163558, reg_alpha=0.01794072813847298, reg_lambda=0.00446283946806131, verbose=-1)
lgbm2 = LGBMClassifier(num_leaves=241, max_depth=11, learning_rate=0.04267548857660626, n_estimators=176, min_child_samples=17, min_child_weight=0.34453075904087466, subsample=0.9420126904183629, colsample_bytree=0.8969001870626069, reg_alpha=0.2534079006185169, reg_lambda=0.22093082929171354, verbose=-1)
lgbm3 = LGBMClassifier(num_leaves=227, max_depth=10, learning_rate=0.02643396947526321, n_estimators=247, min_child_samples=16, min_child_weight=0.3399998175512427, subsample=0.9860913792672888, colsample_bytree=0.9193166643298747, reg_alpha=0.007954215605781124, reg_lambda=0.1780820734696766, verbose=-1)

lgbm1.fit(X_train_transformed, y_train)
lgbm2.fit(X_train_transformed, y_train)
lgbm3.fit(X_train_transformed, y_train)
lgbm1_accuracy = accuracy_score(y_val_transformed, lgbm1.predict(X_val_transformed))
lgbm2_accuracy = accuracy_score(y_val_transformed, lgbm2.predict(X_val_transformed))
lgbm3_accuracy = accuracy_score(y_val_transformed, lgbm3.predict(X_val_transformed))

print(f"Tuned LGBM1 Accuracy: {lgbm1_accuracy:.4f}")
print(f"Tuned LGBM2 Accuracy: {lgbm2_accuracy:.4f}")
print(f"Tuned LGBM3 Accuracy: {lgbm3_accuracy:.4f}")

    Ensemble Learning with Voting Classifier

    Combined above three tuned LightGBM models using a Voting Classifier for better generalization. Achieved an accuracy of 0.7359, slightly lower than the best individual LGBM model (0.7493).

    Despite the lower accuracy, ensembling improves robustness and is expected to perform better on unseen data. And boom — rightly so! I submitted this model to check accuracy on the private data, and it came out as 0.6470. That’s how I landed at #1 on the leaderboard.

    0.6470 Screenshot

    jupyter-notebook.ipynb
    base_models = [('lgbm1', lgbm1), ('lgbm2', lgbm2), ('lgbm3', lgbm3)]

voting_classifier = VotingClassifier(estimators=base_models, voting='hard')
voting_classifier.fit(X_train_transformed, y_train)
voting_classifier_accuracy = accuracy_score(y_val_transformed, voting_classifier.predict(X_val_transformed))

print(f"Voting Classifier Accuracy: {voting_classifier_accuracy:.4f}")

    Tuning 2.0: The Diabolical Tuning

    Why settle for less when you can tweak, tune, and keep trying? I kept adding more LGBMs, always chasing that elusive 0.75 accuracy. Sure, it's a rollercoaster with overfitting on the private data scores, but you know what? This is where the fun begins. After all, in India, we don’t stop at Good Enough.

    Diabolical Laser

    jupyter-notebook.ipynb
    # 5 More LGBM Models
lgbm4 = LGBMClassifier(num_leaves = 181, max_depth = 10, learning_rate = 0.04130232296795349, n_estimators = 228, min_child_samples = 29, min_child_weight = 1.8974552838538845, subsample = 0.7827368388408219, colsample_bytree = 0.7629249518899549, reg_alpha = 0.07533822749656874, reg_lambda = 0.0015430279436605976)
lgbm5 = LGBMClassifier(num_leaves=169, max_depth=10, learning_rate=0.05233350977069071, n_estimators=231, min_child_samples=22, min_child_weight=0.4601062504283119, subsample=0.9390809394397799, colsample_bytree=0.7764703821961773, reg_alpha=0.1598399044304671, reg_lambda=1.6612340944171649)
lgbm6 = LGBMClassifier(num_leaves=269, max_depth=11, learning_rate=0.035393108325162924, n_estimators=229, min_child_samples=23, min_child_weight=0.5340539301261625, subsample=0.9246962331578512, colsample_bytree=0.9088757439239776, reg_alpha=0.2602063481715106, reg_lambda=1.0107331287932972)
lgbm7 = LGBMClassifier(num_leaves=186, max_depth=11, learning_rate=0.04584370337618702, n_estimators=151, min_child_samples=25, min_child_weight=0.1159493172007911, subsample=0.5484014391486206, colsample_bytree=0.7508636391009044, reg_alpha=0.004148047682050917, reg_lambda=0.0072449781860701905)
lgbm8 = LGBMClassifier(num_leaves = 160, max_depth = 13, learning_rate = 0.041619634901885666, n_estimators = 252, min_child_samples = 39, min_child_weight = 0.01163215435201914, subsample = 0.9461943583274226, colsample_bytree = 0.7569519218012518, reg_alpha = 0.003349476296685088, reg_lambda = 0.1693179077289772)

# One XGB Model
xgb = XGBClassifier(n_estimators = 387, max_depth = 9, learning_rate = 0.026577196331014415, subsample = 0.985994351977102, colsample_bytree = 0.8102255771398449, gamma = 1.2078424008928528, reg_alpha = 0.18482880642623628, reg_lambda = 0.03203503173352191, scale_pos_weight = 1.004871302905267)

lgbm4.fit(X_train_transformed, y_train)
lgbm5.fit(X_train_transformed, y_train)
lgbm6.fit(X_train_transformed, y_train)
lgbm7.fit(X_train_transformed, y_train)
lgbm8.fit(X_train_transformed, y_train)
xgb.fit(X_train_transformed, y_train)
lgbm4_accuracy = accuracy_score(y_val_transformed, lgbm4.predict(X_val_transformed))
lgbm5_accuracy = accuracy_score(y_val_transformed, lgbm5.predict(X_val_transformed))
lgbm6_accuracy = accuracy_score(y_val_transformed, lgbm6.predict(X_val_transformed))
lgbm7_accuracy = accuracy_score(y_val_transformed, lgbm7.predict(X_val_transformed))
lgbm8_accuracy = accuracy_score(y_val_transformed, lgbm8.predict(X_val_transformed))
xgb_accuracy = accuracy_score(y_val_transformed, xgb.predict(X_val_transformed))


print(f"Tuned LGBM4 Accuracy: {lgbm4_accuracy:.4f}")
print(f"Tuned LGBM5 Accuracy: {lgbm5_accuracy:.4f}")
print(f"Tuned LGBM6 Accuracy: {lgbm6_accuracy:.4f}")
print(f"Tuned LGBM7 Accuracy: {lgbm7_accuracy:.4f}")
print(f"Tuned LGBM8 Accuracy: {lgbm8_accuracy:.4f}")
print(f"Tuned XGB Accuracy: {xgb_accuracy:.4f}")

    Multi Layer Diabolical Voting

    Because one layer of chaos wasn’t enough. I stacked voting classifiers on top of other voting classifiers — hoping the ensemble madness captures patterns that single models miss. This is ensembling... But on Steroids.

    Voting Classifier : Layer 1

    jupyter-notebook.ipynb
    base_models_2 = [('lgbm8', lgbm8), ('lgbm4', lgbm4), ('lgbm5', lgbm5), ('lgbm6', lgbm6), ('lgbm7', lgbm7), ('xgb', xgb)]

voting_classifier_2 = VotingClassifier(estimators=base_models_2, voting='hard')
voting_classifier_2.fit(X_train_transformed, y_train)
voting_classifier_accuracy_2 = accuracy_score(y_val_transformed, voting_classifier_2.predict(X_val_transformed))

print(f"Voting Classifier 2 Accuracy: {voting_classifier_accuracy_2:.4f}")
    jupyter-notebook.ipynb
    base_models_3 = [('lgbm8', lgbm8), ('lgbm4', lgbm4), ('lgbm5', lgbm5), ('lgbm2', lgbm2), ('lgbm3', lgbm3)]

voting_classifier_3 = VotingClassifier(estimators=base_models_3, voting='hard')
voting_classifier_3.fit(X_train_transformed, y_train)
voting_classifier_accuracy_3 = accuracy_score(y_val_transformed, voting_classifier_3.predict(X_val_transformed))

print(f"Voting Classifier 3 Accuracy: {voting_classifier_accuracy_3:.4f}")

    Voting Classifier : Layer 2

    jupyter-notebook.ipynb
    base_models_4 = [('vc2', voting_classifier_2), ('vc3', voting_classifier_3)]

voting_classifier_4 = VotingClassifier(estimators=base_models_4, voting='hard')
voting_classifier_4.fit(X_train_transformed, y_train)
voting_classifier_accuracy_4 = accuracy_score(y_val_transformed, voting_classifier_4.predict(X_val_transformed))

print(f"Voting Classifier 4 Accuracy: {voting_classifier_accuracy_4:.4f}")

    Final Submission

    Submitted this Monster, and boom — accuracy jumped to 0.6495 (Yes Yes, Let’s call it a respectable 0.65!). And guess what? Still dominating the leaderboard at #1.

    jupyter-notebook.ipynb
    predictions = voting_classifier_4.predict(test_transformed)  

pred_df = pd.DataFrame({
    'id': range(10000),
    'target': predictions
})  

# Save the predictions DataFrame to a CSV file
pred_df.to_csv('submission.csv', index=False)

    0.6495 Screenshot

    Effective Leaderboard

    Model Evaluation

    Malware detection recall is 79%, meaning the model successfully identifies most infected systems but may misclassify some clean ones. False negatives are lower than false positives, indicating the model prioritizes catching malware over mistakenly marking clean systems.

    jupyter-notebook.ipynb
    ConfusionMatrixDisplay.from_predictions(y_val_transformed,voting_classifier_4.predict(X_val_transformed))
plt.show()

    Confusion Matrix

    jupyter-notebook.ipynb
    print(classification_report(y_val_transformed,voting_classifier.predict(X_val_transformed)))

    Conclusion

    1. Signature Version is the most important feature, indicating that malware detection heavily relies on antivirus definitions.
    2. CityID matters, suggesting some cities are highly targeted, which aligns with real-world attack patterns.
    3. DiffOS indicates that users who install or update their system before updating antivirus definitions are more likely to be affected by malware, highlighting a key feature engineering direction.
    4. NumericOSVersion, AntivirusConfigID impact detection, as malware may exploit outdated or misconfigured OS versions.
    jupyter-notebook.ipynb
    feature_importance = pd.DataFrame({
    'Feature': column_transformer.get_feature_names_out(),
    'Importance': lgbm1.feature_importances_
})

feature_importance = feature_importance.sort_values(by='Importance', ascending=False)

feature_importance[:10]

    If you made it this far — you're a real one. Whether you're here for the ML, the memes, or just lurking for leaderboard secrets, I hope this ride was worth your time.

    This wasn't just about winning — it was about learning, experimenting, overfitting like a maniac, and still shipping something that worked.

    Thanks for Surviving This Madness