Open this if you want to see how ChatGPT built all my code.

Disclaimer:

I build Machine Learning models for a living. I know what questions to ask, and I can spot mistakes in ChatGPT's suggestions.

ChatGPT is an incredible tool, but you must understand how to use it properly.

Let's get started!

I want to build a neural network to solve the MNIST problem using Keras.

I asked ChatGPT to start things off for me.

The result worked flawlessly without having to change anything.

This was a great start!

ChatGPT used Categorical CrossEntropy as the loss function. I asked it to use a different one.

The updated code also works.

Notice how it changed the loss function, but it also dropped the categorical transformation of labels. This is key for the new loss function work!

ChatGPT's code uses fully-connected layers.

But I want to use convolutional layers.

ChatGPT modified the code and added a Conv2D and a MaxPooling layer.

I need to evaluate the final model, so I asked ChatGPT to write code using the test set.

The result and explanation look great!

The model uses the entire test set as validation data during training.

Let's ask ChatGPT to change that to 20% of the train data.

This is good stuff!

(By the way, every answer from ChatGPT comes with a full explanation. I'm not including it here to keep screenshots simple.)

I want smaller batches and run the code for fewer epochs.

Of course, I don't need ChatGPT for this, but I did not want to change my code without updating ChatGPT's context. That's why I asked.

Let's now draw a plot of the training and testing losses during the training process.

ChatGPT explained that I need to change the line that fits the model to capture its result value (history.) That was neat!

The plot works and looks good!

I need to show a few examples from the dataset, so let's ask ChatGPT to write code to print a combination of images and labels.

The code was also perfect.

You can see the plot with 20 images attached.

Something useful whenever you build a model is to take a look at its structure.

Let's ask ChatGPT how I can do that.

Okay, the summary of the model is helpful, but I'd rather see a plot showing the model's structure.

Let's ask:

This is looking good! Let's now get ready to deploy this model.

Let's ask ChatGPT to save the model to disk.

Now, I want to create a class that uses the saved model to make predictions.

This is an interesting prompt, and the solution is perfect.

Let's now write an example that uses the Predictor class to predict the label of 10 random images:


To finish this, let's display a confusion matrix.

I like the styles that ChatGPT used!

Here is a link to all of the code that ChatGPT produced:

https://colab.research.google.com/drive/1JX1AVIfGtIlnLGqgHrK6WPylPhZvu9qe?usp=sharing

This is going to be such an exciting year!

If you enjoy content like this, follow me @svpino.

ChatGPT built all of my code

11 ways ChatGPT saves me hours of work every day, and why you'll never outcompete those who use AI effectively.

A list for those who write code:

1. Explaining code

Take some code you want to understand and ask ChatGPT to explain it.

I've found explanations are very detailed. This is much quicker than trying to figure out convoluted code.

2. Improve existing code

Ask ChatGPT to improve existing code by describing what you want to accomplish.

It will give you instructions about how to do it, including the modified code.

3. Rewriting code using the correct style

This is great when refactoring code written by non-native Python developers who used a different naming convention.

Notice how ChatGPT not only gives you the updated code; it also explains the reason for the changes.

4. Rewriting code using idiomatic constructs

Very helpful when reviewing and refactoring code written by non-native Python developers.

ChatGPT knows the "Pythonic" way, and it will give you suggestions to improve your code and make it much more readable.

5. Simplifying code

This is one of my favorite tricks: Ask ChatGPT to simplify complex code.

The result will be a much more compact version of the original code.

Notice the explanation and how it tells us this is simpler but not the most efficient.

6. Writing test cases

This has become one of my favorite ChatGPT abilities: Ask it to help you test a function, and it will write test cases for you.

This example focuses on the quick_sort function from the previous example.

7. Exploring alternatives

ChatGPT told me its Quick Sort implementation wasn't the most efficient, so I asked for an alternative implementation.

This is great when you want to explore different ways to accomplish the same thing.

I've learned a ton from this!

8. Translating code

Anytime you want to port some code from one language to another, ask ChatGPT to help you.

9. Writing documentation

This is another one of my favorite tricks.

Ask ChatGPT to write the documentation for a piece of code, and it usually does a great job.

It even includes usage examples as part of the documentation!

10. Tracking down bugs

If you are having trouble finding a bug in your code, ask ChatGPT for help.

It took ChatGPT seconds to find the bug in the attached example. I don't know about you, but it would have taken me much longer than that.

11. Scaffolding

Probably the way I use it the most is to kick off the structure of anything new I want to write. GitHub Copilot does a great job at this as well.

An example where this is very useful is when interacting with a RESTful API.

Something to keep in mind:

I have 2+ decades of programming experience. I like to think I know what I'm doing.

I don't trust people's code (especially mine,) and I surely don't trust ChatGPT's output.

This is not about letting ChatGPT do my work. This is about using it to 10x my output.

ChatGPT is flawed. I find it makes mistakes when dealing with code, but that's why I'm here: to supervise it.

Together we form a more perfect Union. (Sorry, couldn't help it)

Developers who shit on this are missing the point.

The story is not about ChatGPT taking programmers' jobs. It's not about a missing import here or a subtle mistake there.

The story is how, overnight, AI gives programmers a 100x boost.

Ignore this at your own peril.

Next time, I'll show you how to build an end-to-end deep learning model using ChatGPT.

I'll only ask questions.

It'll blow your mind.

Follow me @svpino to ensure you don't miss it.

11 ways you can use ChatGPT to write code

The first problem you face when starting with machine learning:

Finding the perfect balance between training for too long or not training enough.

Either one and your model will be trash.

Here is the simplest and one of the most effective ways to work around this:

First of all, remember this:

• Overfitting will likely happen if you train for too long.
• Underfitting will likely happen if you don't train long enough.

It's not about time, but about number of iterations—or epochs.

We need a way to find the correct number.

First idea:

Use a separate holdout set to evaluate how the model is doing while we train it.

We can evaluate it after each iteration and stop the process as soon as the model's performance degrades.

We want to stop at the sweet spot in the attached chart.

Notice that the chart shows the "error" of the model.

We call this "loss."

We want the holdout validation set and a metric (in this case, the validation loss.) We can also use any metric to determine when overfitting starts happening.

But there's a problem:

I don't want to draw a chart and find the sweet spot every time I need to do this.

I don't want to do any manual work.

How can we stop the model automatically when it starts overfitting?

Let's think about it:

The loss might be noisy: it will oscillate up and down.

We can't stop the process the first time the loss increases because it might be too early.

We need to wait for a few consecutive values going in the wrong direction before stopping the process.

One last problem:

Let's say we wait for the loss to increase for 5 consecutive epochs before we stop training.

At that point, our model has been overfitting for 5 epochs. We don't want that!

We want the model we had 5 epochs ago. 

How do we do this?

As we train the model, we can save the model as long as its validation loss is better than the last copy we saved.

• Run one iteration
• If the loss is better, save a copy

If we do this, we will have the best model saved and ready to use when we stop the process.

This process is called Early Stopping and every major library supports it.

The three ingredients we need to make it happen:

• A holdout set 
• A metric
• A trigger with patience
• A way to save the best model

Twitter is great, but nothing beats YouTube if you want to understand a topic.

I recorded a YouTube video that goes into more details:

https://youtu.be/CODw8292uqE

Every week, I break down machine learning concepts to give you ideas on applying them in real-life situations.

Follow me @svpino to ensure you don't miss what's coming next.

Early stopping

This is a step-by-step guide to building your first deep learning model.

Let's get started:

Building the model is essential, but it's just the beginning.

I want to give you everything you need to understand what's going on:

• A way to make changes
• A way to experiment
• A way to keep track of everything you did

We are going to use a neat tool for that.

We are going to use @Cometml to keep track of our experiments.

Create a free account here: https://www.comet.com/signup?utm_source=svpino&utm_medium=referral&utm_campaign=online_partner_svpino_2022

You can also connect your GitHub account. The process will take 10 seconds.

To write the model, I will use Google Colab, which is also free.

You can open it here: https://colab.research.google.com/.

Create a new notebook, and let's get started.

I'll share my notebook with my code at the end of this post.

We will start by installing @Cometml, initializing it, and defining what I want from my experiments.

You'll need your API Key from your account under settings.

This is all we need for @Cometml to do its magic. You'll see it at the end of this post.

Right after that, we'll import every module we need.

This is boring but necessary.

We will use the MNIST dataset to build this model.

Our goal:

Given an image we haven't seen before, the model should determine the handwritten digit it shows.

The attached line loads the data into a train and test set.

Instead of working with pixel values from 0 to 255, we will scale them to fit between 0 and 1.

Models are much happier when we scale the data we feed them.

It's nice to see what we've done so far.

Here are a few of the handwritten digits.

I wrote some code to display them. You'll find it in my notebook at the end of this thread.

One more thing before we are done with the images:

These digits are grayscale, so they only use one channel for their color.

Unfortunately, the shape of the data needs an extra dimension to represent colors (even though there's no color.)

Let's do that.

Time for the high-level structure of the model:

1. Input. 28x28
2. A Conv2D layer using ReLU
3. A MaxPooling layer
4. A fully-connected layer using ReLU
5. An output layer using softmax

You will find a complete explanation of each decision in the notebook at the end.

Now that we have a model, let's define how we want to train it:

1. Stochastic Gradient Decent optimizer
2. Learning rate = 0.01 with 0.9 momentum
3. Sparse Categorical Cross-Entropy as the loss
4. We will track accuracy

Here is what the summary of the model looks like:

You can see the layers, their output shape, and their number of learnable parameters.

Let's train the model.

This example is simple. Even without access to a GPU, it will train fast.

But if you are using Colab, you will be fine.

At this point, you should have a model that scored above 99% accuracy on the train set.

This is cool, but maybe our model is cheating and memorizing all of the images.

Let's make sure everything is working correctly.

Let's show the model images that it hasn't seen before. 

Let's use the test set.

I ran 10,000 images from the test set through the model. Its accuracy was 0.9849.

I'd say this is pretty good.

The last line on the attached picture logs the confusion matrix on @Cometml.

Now go to your @Cometml account and click on "Uncategorized Experiments."

Every time you run your notebook, you'll get a new experiment.

Here are some of the things that @Cometml tracks for you:

Logging experiments is critical. 

Every time you make a change, you'll get a new experiment.

You can now compare and roll back to a different version. You can do a complete analysis of any past experiment.

Everything is there for you. 

So, what's next?

First, let me share the code with a complete explanation of everything I did:

https://colab.research.google.com/drive/1eY1oBKaknSTwdCesLLozwXepFGO_gTUn?usp=sharing

Duplicate this notebook so you can make changes.

Your homework:

Make sure you understand every line of code and decision I made.

Best way to learn: 

Make small changes, look at the results, and compare them with your previous experiments.

Step by step guide to building your first deep learning model

A lesson some people learn too late:

Building a machine learning system is not like building regular software.

But unless you've done it before, you'd think it's all pretty much the same.

Here is a reason they are different and what you can do about it:

For the most part, regular software looks like this:

• Build once
• Run forever*

I know there's no such thing as *forever*. Things change all the time.

But this rate of change pales in comparison with machine learning systems. Here is an example:

A real, straightforward example:

You build a deep learning model to process pictures that users capture with their phones.

The goal of the model: recognize different shoe brands in the pictures. 

Everything works, but something happens:

Apple releases a new iPhone.

A beautiful, groundbreaking new camera that is now packing pixels differently.

The pictures look stunning, but the model starts having problems.

Guess what happened?

We trained our model on different pictures.

The world changed, but our model didn't. 

Although the new pictures aren't enough reason to break our model, there're enough differences to reduce its performance.

Unfortunately, there's more:

Nike just released a new pair of shoes. Adidas did the same.

Both go viral. Everyone is buying them!

When we trained our model, these shoes didn't exist. We didn't know about them, so our model doesn't either.

The model's performance takes another hit.

It's been a few weeks. We had an excellent system that turned into a mediocre one.

The model is the same. The world isn't.

In machine learning, we call this problem "drift." 

Specifically, I gave you examples of data and concept drift.

It's natural for those who haven't built complete systems before to assume their work is done as soon as they finish training their model.

In reality, that's when the work truly starts.

Building a model is simple. Building a model that works is another ball game.

Interestingly, most people know how to tackle drift. 

But they are missing a critical step: They realize that drift happens when it's too late. 

When everything breaks, they scramble and fix things. 

You need to do better. You need good *monitoring*.

Monitoring helps you see problems coming.

More often than not, it gives you time to react before your system breaks.

I've tried to implement a monitoring system myself.

I'd rather eat glass, so I have a recommendation for you:

Look into @cometml's Model Production Monitoring:

https://www.comet.com/docs/v2/guides/model-production-monitoring/mpm-overview/?utm_source=svpino&utm_medium=referral&utm_campaign=online_partner_svpino_2022

You have access to 3 fundamental levers:

1. Identify if your model is struggling
2. Track drift across inputs and outputs
3. Get alerts if anything is wrong

@Cometml gives you what you need to answer two questions:

1. Am I going to have a problem with my model?
2. Do I already have a problem with my model?

Not having a robust monitoring system is like driving a car without a gas gauge.

For those who prefer to build everything in-house:

A poorly designed monitoring system is worse than no system at all.

A broken alarm at home that doesn't go off when the thieves come in will make you think everything is fine.

Until it's too late.

Model monitoring

Everyone knows they need to replace missing values in their dataset.

Most people, however, miss one critical step.

Here is what you aren't doing and how you can fix it:

I'll start with an example.

A company surveys a bunch of people.

Some people leave one particular question unanswered.

When we collect the data, and before using it to build a model, we must take care of these missing values.

For simplicity's sake, let's assume that replacing missing answers with the mean of all the other answers is a good approach.

We can do that and solve the problem.

Most people stop here. That's their mistake.

Having participants not disclose the answer to a question could be as important as the answer itself!

We want to know who didn't answer the question.

Imputing the data will completely erase that information from the dataset.

You should do something else instead:

Before you replace missing values, create a new column.

You'll flag every participant that didn't answer the question.

In other words, the new column will have a value of 1 if the row has a missing answer and 0 otherwise.

Then you'll impute the missing values.

This extra column will tell the model who didn't answer. 

That information may be necessary. It may also be useless.

There's only one way to find out.

I wanted to summarize this post by reinforcing the need for that extra column.

But I think I have better advice:

Make sure you understand where your data is coming from and the reason you have missing values in the first place.

I recorded a YouTube video talking about this topic for those who prefer video:

https://www.youtube.com/watch?v=f9AQy7p0QEo

The issue with replacing missing values in your dataset

I'm sure you've used data augmentation before. 

Most people think of data augmentation as a technique to improve their model during training.

This is true, but they are missing something.

Here's a brilliant approach to improve the predictions of your model:

Let's start with a quick definition of data augmentation:

Starting from an initial dataset, you can generate synthetic copies of each sample that will make a model resilient to variations in the data.

This is awesome. But there's more we can do with it.

Imagine you are building a multi-class classification model. 

When testing it, you do this:

• Take every sample
• Run it through the model
• Determine the correct class from the result

You have one opportunity to make the correct prediction.

But you can do better:

You can take advantage of data augmentation to give you a better opportunity to make the correct prediction.

Introducing "Test-time augmentation."

You can augment samples before running them through the model, then average the prediction results.

Imagine you are ready to make a prediction.

Instead of running that sample through the model, you generate three versions of it by changing the image's contrast, rotating it slightly, and cropping it a bit.

You now have four different samples.

Run all four images through the model, average the softmax vectors you get back, and determine the final class from the result.

By augmenting the original image, you give the model more opportunities to "see" something different and compute the correct prediction.

The success of Test-time augmentation depends on how good are your augmented samples.

That's where most of your time will go.

Every augmented image will have a lot of influence on the final result. You don't want sloppy variations!

I've found a lot of success using a few slight modifications to the initial picture. 

"Less is more" in this process.

The best part: It's a relatively easy way to improve your predictions.

Test-Time Augmentation

I've helped hundreds of people start with machine learning.

Everyone asks me the same, fundamental question.

But they all hate my answer. Engineers even more.

Let me try again, but this time I'll show you a few lines of code that will 10x your process:

People always ask: "How do you know what to do now?"

The answer is simple, but nobody likes to hear it: "Well, you don't know."

After a few seconds, I follow up: "You need to experiment to find what works best."

Here is the reality they don't want to hear about:

Machine learning models are hard to optimize for any given problem. 

There are just too many variables that we could change!

People aren't used to this. It's hard for them to move from "I know what will happen" to "I need to try and see."

Here is one example:

Let's talk about a concrete idea: "hyperparameters."

Think of these as "configuration settings." Depending on the values you choose, your model will perform differently.

These are the knobs and levers of every machine learning model.

Every model has different hyperparameters. 

Here are a few common examples:

• learning rate
• batch size
• epochs
• regularization

The list goes on and on.

In a perfect world, we will find the combination of values for these hyperparameters that's ideal for our problem.

But finding these values is a headache. There are too many possibilities!

We need something better.

I consider myself an organized person.

When I started building models, I kept a spreadsheet with different combinations of values.

Whenever I tried a set of hyperparameters, I carefully took notes of values and results.

There are two problems with this.

First problem: 

You don't want to run your experiments manually. There's little chance you'll find an optimal combination by guessing.

Second problem: 

Writing down values in a spreadsheet is cumbersome, suboptimal, and prone to errors.

To solve the "finding hyperparameters by hand" part, these are the libraries I use:

1. KerasTuner
2. Optuna

With these, you can automatically find the best hyperparameters for your model.

For those who like code, here are two examples:

The first example uses KerasTuner to train a neural network on the Penguins dataset: https://deepnote.com/@svpino/Keras-Tuner-z31YNDzcSQKAqJGZSqTDPQ

The second example uses Optuna to optimize an XGBRegressor and a CatBoostRegressor: 
https://deepnote.com/@svpino/Tuning-Hyperparameters-with-Optuna-6hoSPY0vTiCPIpXwdHDVVw

The first problem was running multiple experiments to find the best set of hyperparameters.

But how do you keep track of all of these experiments?

You can't rely on spreadsheets, at least not if you want to keep your sanity.

Here is where @Cometml enters the picture.

With a few lines of code, you can synchronize your work and keep track of everything you do.

Here is the same example code as before, but this time using @Cometml:

https://deepnote.com/@svpino/Keras-Tuner-CometML-f60a5301-1ad2-4b97-babc-7c7e12f1b45f

A couple of notes:

Notice that my code stayed the same. I just added a couple of lines to connect to @Cometml.

I get all sorts of metrics and charts right from @Cometml's interface. All of them for free!

Over time, you'll have a library of experiments with detailed metrics about everything you did. 

You can compare experiments, reproduce them, filter, or go back to any one of them. 

You won't ever go back to manual tracking. I promise.

Here is the link I want you to bookmark:

https://www.comet.com/docs/?utm_source=svpino&utm_medium=social&utm_campaign=online_paii_2022

Find some sample code, and click on the corresponding Quickstart guide on this page.

It's a long list of supported libraries, so you'll find something that works for you.

Every week, I post 1 or 2 threads like this, breaking down machine learning concepts and giving you ideas on applying them in real-life situations.

Follow me @svpino and make sure you don't miss my next thread.

Hyperparameters

One of the most common problems in machine learning:

How do you deal with imbalanced datasets?

Not only does this happen frequently, but it's also a popular interview question.

Here are seven different techniques to deal with this problem:

What's an imbalanced dataset?

Imagine you have pictures of cats and dogs. Your dataset has 950 cat pictures and only 50 dog pictures.

That's an imbalanced dataset. 

There's a significant difference in the number of samples for each class.

Why is an imbalanced dataset a problem?

A model that classifies every picture as a cat will be 95% accurate!

Think about this: A dumb model will get you to 95% accuracy because of your imbalanced classes.

That's a big problem, and here is how you solve it:

1. Accuracy is not a good metric for imbalanced problems.

Instead, look at any of the following metrics:

• A combination of Precision and Recall
• F-Score
• Confusion Matrix
• ROC Curves

2. Collect more data.

If you can find more dog pictures, do that. 

Sometimes this is not possible, but other times it's the simplest solution you can do.

3. Augment the dataset with synthetic data.

If you have the means to create realistic samples, you can do that to augment the dataset and balance it.

For example, Tesla uses synthetic data to train their models on non-common situations.

4. Resample your dataset.

• Oversample the pictures of dogs.
• Undersample the pictures of cats.

You can also combine both.

Here is an example:

You can resample our hypothetical dataset by doing the following:

• Use every dog picture four times.
• Use every other cat picture.

New dataset:

• Dogs: 400 pictures (50 × 4)
• Cats: 475 pictures (950 ÷ 2)

Much better, huh?

Important note:

Both over and undersampling introduce biases into your dataset. You are changing the data distribution by arbitrarily messing with the existing samples.

Make sure you keep this in mind and think about the consequences.

Let's continue:

5. Weight each class differently.

There are multiple techniques to weight each class differently and have the model pay more or less attention to those samples.

For example, we can have a large weight for dogs to compensate for the lack of samples.

6. Different algorithms handle imbalances differently.

Decision Trees are excellent at handling imbalanced classes. Neural networks, not so much.

Make sure you use the correct algorithm to work on your problem.

7. Make sure you approach the problem correctly.

Many people have tried to solve anomaly detection problems using multi-class classification.

That's the wrong approach. 

Understand what problem you are trying to solve before deciding how to do it.

Let's recap how you can handle an imbalanced dataset:

1. Pick the appropriate performance metric
2. Collect more data
3. Generate synthetic data
4. Resample the dataset
5. Use different weights
6. Try different algorithms
7. Approach the problem correctly.

Dealing with imbalanced datasets

Many people new to machine learning have no idea that labeling data is a problem they need to think about.

To be clear: "labeled datasets" aren't a thing in the real world.

Here is an excellent approach to getting past this problem:

I've found that gaining access to lots of data is not always an issue.

But getting access to "labeled data" is usually not a thing.

Sometimes you can solve this by throwing people at the problem, but this is not always a solution.

Looking for cats on images? Labeling will be relatively cheap.

Looking for cancer in x-rays? Good look finding enough specialists to work with you. 

Even worse: imagine that "labeling" means having to drill a hole to determine whether there's oil underground.

If you don't have reliable labels, you can't use Supervised Learning techniques.

How do you move past this?

Let's talk about Active Learning.

Active Learning is a semi-supervised learning method.

Bottom line: we can start building a model with a few labeled samples.

Not zero, but not the entire dataset.

We will start training with a few samples and interactively ask for new labeled data as we need it.

The critical idea here:

We will only need to label the most informative samples.

Instead of needing 10,000 labels in a 10,000-sample dataset, we will only need a few of them: the essential ones that will maximize the learning of our model.

Here is a rough, hypothetical sketch of how it works:

• Start with 1,000 labeled samples
• Train a model
• Use that model to predict the 9,000 samples
• Label 10% of the worst predictions
• Repeat

I mentioned above that "worst predictions" are the samples that will be the most informative.

But how do you measure that?

You can use multiple heuristics to determine which samples to label next.

Here are some existing methods that you can use:

• Least Confidence Uncertainty
• Smallest Margin Uncertainty
• Entropy Reduction

If you are interested, do some research on each of these.

To recap:

We can use Active Learning to train a model while minimizing the amount of labeled data.

This translates into significant savings. Sometimes, this is the difference that could make your solution viable.

This post is a good reflection of the content I post weekly.

Follow me @svpino for practical tips and stories as I deconstruct my experience as an engineer focused on applied machine learning.