Open in app

Sign in

Write

Sign in

A Practice of Fabric Defect Detection by Using CNN

Isada Sukprapa
13 min readMay 14, 2020

--

We demonstrate Fabric Defect Detection using CNN to provide general knowledge about implementing CNN

Fig 1: The fabric photo by Engin Akyurt. Source

Introduction

Recently there has been a rise in popularity for edge-based AI implementation, which is an implementation of AI outside of the main computing unit to lessen load requirements of the central server and help out in some smaller computation processes such as preprocessing or classification using pre-trained models. An example of edge-based AI implementation is defect detection using edge devices where the central server does not necessarily need to know what went wrong with each unit in real-time. Directly receiving image feed from the production line and looking for defects will take away a significant amount of traffic from the server, and thus edge implementation to detect defects will lessen this load. While AI is not considered a new technology nowadays, and neither do IoT devices, we will be attempting to implement a rudimental defect detection using CNN in this article.

Fig 2: Convolutional Neural Network (CNN). Source

In 1998, Yann LeCun, et al. published the Convolutional Neural Network (CNN) for the first time in the journal paper titled “Gradient-Based Learning Applied to Document Recognition.” Thenceforth, the CNN model has been considered as the most powerful deep neural network, which is mostly used for analyzing images.

Presently, defect detection in manufacture is based on Deep Learning. For instance, D. Hou et al., 2019 implemented laser chip defect detection using CNN based on GoogleNet with an accuracy of 97 percent. Additionally, K. Lee et al., 2019 demonstrated the comparison among the three techniques: SVM, DNN, and CNN, resulting in CNN as the best performer with 95.1 percent accuracy.

Therefore, CNN is considered the best option for defect detection usage on our dataset, which will be a set of monotone images of fabric labeled as having defects and not having a defect.

Dataset

The dataset is provided by J. Silvestre-Blanes et al., 2019, in the paper “A Public Fabric Database for Defect Detection Methods and Results.” The dataset has 245 images with seven different fabrics. There are 140 non-defect images and 20 for each type of fabric. With different types of defects, there are 105 images. Images have a size of 4096×256 pixels.

A sample of the defect image

# Load picture
pic = project_path + “/dataset/Defect_images/0003_002_00.png”
im = imageio.imread(pic)
print(“picture shape: {}”.format(im.shape))
print(“picture matrix: \n {}”.format(im))
# Set display size
plt.figure(figsize=(20,10))
plt.imshow(im, cmap="gray")
plt.show()

In this practice, we would like to classify only the two classes, which are defect-free and defected, based on the CNN technique.

Note that the project_path is our project directory

Implementation

In the upcoming steps, we will be elaborating on how to implement the fabric defect classification according to the CNN model.

Prerequisite

Let us define the libraries and tools that have been used.

Dependency

In this article, we will be using CNN to identify defects on clothing surfaces. Moreover, here are the dependencies:

python >= 3.6
tensorflow >= 2.2.0
pandas >= 1.0.3
glob >= 0.7
imageio >= 2.4.1
matplotlib >= 3.2.1
sklearn >= 0.22.2

Optional but recommended

Google Colaboratory is a free tool that provides powerful resources such as a GPU, accelerating deep learning training processes. Besides, the module installing process is easy and similar to Jupyter Notebook, that is,

!pip install <package>

All libraries used in this article are provided in Google Colaboratory

Model Training Procedure

  1. Loading External File

In the first step, we have defined the load_file function to read image files. The function will return a NumPy array of the image object.

def load_file(file_path, label):
    # Include the project path
    file_path = project_path + file_path
    
    # Declare the folder name
    folder_name = file_path.split("/")[-1]
    
    # Declare output list
    out_list = []
    
    # Load every file .png format
    for image_path in glob.glob(file_path + "/*.png"):
        # Read image file
        image = imageio.imread(image_path)
        
        # Declare temporary dict dtype
        temp = {}
        
        # Set the file name
        temp["name"] = image_path.split("/")[-1]
        
        # Set the file label, 0 for non defect. 1 for defect
        temp["label"] = label        # There are somes images are tensor dtype
        # Thus I fix by selecting only a tensor index zero
        try:   
            temp["data"] = image[:,:,0].astype("int") 
        except:
            # Normal case
            temp["data"] = image.astype("int")
        
        # Append temp into output list
        out_list.append(temp)
    
    # Print process status by checking size of output list
    if len(out_list) == 0:
        print("loading files from folder: {} is   failed".format(folder_name))
    else:
        print("loading file from folder: {} is successful".format(folder_name))
    
    # Convert list into numpy array dtype
    return np.array(out_list)

The next step is to load the .png file by calling the previous function:

# Define the path
defect_images_path = "/dataset/Defect_images"
non_defect_images_path1 = "/dataset/NoDefect_images/2306881-210020u"
non_defect_images_path2 = "/dataset/NoDefect_images/2306894-210033u"
non_defect_images_path3 = "/dataset/NoDefect_images/2311517-195063u"
non_defect_images_path4 = "/dataset/NoDefect_images/2311694-1930c7u"
non_defect_images_path5 = "/dataset/NoDefect_images/2311694-2040n7u"
non_defect_images_path6 = "/dataset/NoDefect_images/2311980-185026u"
non_defect_images_path7 = "/dataset/NoDefect_images/2608691-202020u"
mask_images_path = "/dataset/Mask_images"defect_images = load_file(defect_images_path, 1)
non_defect_images1 = load_file(non_defect_images_path1, 0)
non_defect_images2 = load_file(non_defect_images_path2, 0)
non_defect_images3 = load_file(non_defect_images_path3, 0)
non_defect_images4 = load_file(non_defect_images_path4, 0)
non_defect_images5 = load_file(non_defect_images_path5, 0)
non_defect_images6 = load_file(non_defect_images_path6, 0)
non_defect_images7 = load_file(non_defect_images_path7, 0)
mask_images = load_file(mask_images_path, -1)# contribute the non defect dataset into one file
non_defect_images = np.concatenate((non_defect_images1, non_defect_images2))
non_defect_images = np.concatenate((non_defect_images, non_defect_images3))
non_defect_images = np.concatenate((non_defect_images, non_defect_images4))
non_defect_images = np.concatenate((non_defect_images, non_defect_images5))
non_defect_images = np.concatenate((non_defect_images, non_defect_images6))
non_defect_images = np.concatenate((non_defect_images, non_defect_images7))

Here, the size of the dataset:

print(“defect_images.shape: {}\nnon_defect_images.shape: {}\nmask_images.shape:{} \n”.format(defect_images.shape, non_defect_images.shape, mask_images.shape))

The dataset also includes defect mask images which are binary images showing a specific defect location in the dataset. Other classification methods could use this, but this will not be utilized in the article.

2. Data preparation

In the stage of training the model, it remaps the image dataset onto a graph

with each axis representing these aspects:

  • X is a vector that each row that consists of an image matrix.
  • y is a matrix label where 0 be a defect-free image and 1 be a defect image.

To balance the dataset, because the number of defect images can not be increased, thus, 105 defect-free images have been selected to equal the number of defect images.

# We random by shuffling the order of defect-free and defect images
np.random.shuffle(non_defect_images)
np.random.shuffle(defect_images)# The class size is selected by the min of length compared between 
# defect-free and defect images
class_size = defect_images.shape[0] if defect_images.shape[0] <= non_defect_images.shape[0] else non_defect_images.shape[0]# Declare dataset by concat defect_images and non_defect_images 
# with length from 0 until class_size
dataset = np.concatenate((defect_images[:class_size], non_defect_images[:class_size]))# Create an empty matrix X with is matrix of 256x4096
# with row equals dataset's length 
X = np.empty([dataset.shape[0], 256, 4096]).astype(int)# Create vector y that has dataset length
y = np.empty(dataset.shape[0]).astype(int)# Assign the X,y one-by-one
for i in range(dataset.shape[0]):
    X[i] = dataset[i][“data”]
    y[i] = dataset[i][“label”]# Since Keras acquire the Image input is a tensor type, we reshape X
X = X.reshape(X.shape[0], 256, 4096, 1)# Display size of the label 0 and label 1 
np.unique(y, return_counts=True)

Now, the size of the dataset is:

(array([0, 1]), array([105, 105]))

As a show, our dataset has a size equal to 210 images, containing 105 defect-free and 105 defect images. In general, the Deep Learning model acquires an amount of dataset (~10K). Therefore, the 210 dataset number is not decent and may cause low accuracy.

3. Create the CNN model

There are many ways to approach the CNN architecture design. We have chosen to use AlexNet (A. Krizhevsky et al., 2012) as the basis and starting with five convolution layers and three fully connected layers. During implementation, the architecture was fine-tuned for optimization.

Fig 3: AlexNet. Source

In finally, our model consists of 7 layers which include:

  • 1st: Convolution-layer with 16 filters 7x7 stride 2x2 max-pooling
  • 2nd: Convolution-layer with 32 filters 5x5 stride 2x2 max-pooling
  • 3rd: Convolution-layer with 64 filters 3x3 stride 2x2 max-pooling
  • 4th: Convolution-layer with 64 filters 3x3 stride 2x2 max-pooling
  • 5th: Fully-connected-layer with 64 nodes
  • 6th: Fully-connected-layer with 64 nodes
  • 7th: Fully-connected-layer with one node

The last layer is the model output. Each layer primarily uses ReLu as the activation function since it does not cause the vanishing gradient problem, except the output layer prefers using the sigmoid function. Furthermore, we avoid overfitting by adding dropout and batch normalization. For model Compilation, Stochastic Gradient Descent (SGD) was set as an optimizer and Binary Cross-Entropy as the loss function based on computational time.

from tensorflow.keras import datasets, layers, models, optimizers, regularizers
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpointdef create_model(image_shape=(256, 4096, 1), print_summary=False):
    # Initial model
    model = models.Sequential()    # CONV layer: filter 16, stride 7x7
    model.add(layers.Conv2D(16, (7, 7),input_shape=image_shape))
    model.add(layers.BatchNormalization())
    model.add(layers.Activation("relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Dropout(0.25))    # CONV layer: filter 32, stride 5x5
    model.add(layers.Conv2D(32, (5, 5), padding=”same”))
    model.add(layers.BatchNormalization())
    model.add(layers.Activation("relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Dropout(0.25))
 
    # CONV layer: filter 64, stride 3x3
    model.add(layers.Conv2D(64, (3, 3), padding=”same”))
    model.add(layers.BatchNormalization())
    model.add(layers.Activation("relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Dropout(0.25))
 
    # CONV layer: filter 64, stride 3x3
    model.add(layers.Conv2D(64, (3, 3), padding=”same”))
    model.add(layers.BatchNormalization())
    model.add(layers.Activation("relu"))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Dropout(0.25))
 
    model.add(layers.Flatten())
 
    # Fully connected layer -> nn layer with 64 nodes
    model.add(layers.Dense(64))
    model.add(layers.BatchNormalization())
    model.add(layers.Activation("relu"))
    model.add(layers.Dropout(0.25))    # Fully connected layer -> nn layer with 64 nodes
    model.add(layers.Dense(64))
    model.add(layers.BatchNormalization())
    model.add(layers.Activation("relu"))
    model.add(layers.Dropout(0.25))
 
    # Output layer
    model.add(layers.Dense(1, activation="sigmoid"))
 
    # Set model compiler
    model.compile(optimizer="SGD", loss="binary_crossentropy",          metrics=["accuracy"])
 
    # Show the CNN model detail
    if print_summary:
        model.summary()
    
    return model

The summary of the model is:

Layer 1st to Layer 3rd
Layer 4th to Layer 6th

For a description of each layer,

  • Conv2d layer. Convolution layer
  • Max pooling layer. To reduce the size of the matrix.
  • Activation layer. The output function for the previous layer
  • BatchNormalization layer. Normalize by re-centering and re-scaling
  • Dropout layer. To avoid overfitting.
  • Dense layer. The fully connected layer
  • Flatten layer. Flatten the convolution layer into a vector.

4. Train and Evaluate the CNN model

To efficiently measure the model performance, we use the stratified k-fold method, where the algorithm consists of test and train cycles as follows:

Fig 4: Stratified k-fold Partition. Source

For stratified k-fold, each fold separates the dataset to be two groups:

  1. Training dataset. Use in the CNN mode training.
  2. Validate dataset. Evaluate the model at the end of a training epoch.

In each iteration, we have to train the model several times. For simplifying the process, let us write a train function:

def train_model(model, xtrain, ytrain, xval, yval, n_epoch, batch_size):
    # Train CNN model
    # Batch size to reduce memory usage
    # Set early stopping to avoid overfitting
 
    earlystopping = EarlyStopping(monitor="val_accuracy", patience=2)    history = model.fit(xtrain, ytrain, epochs=n_epoch, batch_size=batch_size, validation_data=(xval, yval), callbacks=[earlystopping])
    return history

During training, we use the callback function named earlystopping to halt the training process when the model does not improve. We set a parameter with a two epochs patience threshold, meaning if the validation accuracy cannot increase in 2 iterations, the training process will automatically stop. The function helps us save time.

Now, we are ready to evaluate the model:

from sklearn.model_selection import StratifiedKFold# Set number of split
kfold_splits = 4
# Set number of epoch
n_epoch = 10
# Set batch size
batch_size = 10# Create StratifiedKFold
skf = StratifiedKFold(n_splits=kfold_splits, shuffle=True)for index, (train_indices, val_indices) in enumerate(skf.split(X, y)):
    print(“Training on fold {}/{}…”.format(index+1, kfold_splits))
    
    # Declare x train and x validate
    xtrain, xval = X[train_indices], X[val_indices]
    # Declare y train and y validate
    ytrain, yval = y[train_indices], y[val_indices]    # Print number of class portion
    print(“ytrain: number of samples each class: {}”.format(np.unique(ytrain, return_counts=True)))
    print(“yval: number of samples each class: {}”.format(np.unique(yval, return_counts=True)))
    
    # Clear the model
    model = None
    # Create cnn model
    model = create_model()    print(“Training new iteration on {} training samples, {} validation samples, this may be a while…”.format(xtrain.shape[0], xval.shape[0]))
 
    # Train CNN model
    history = train_model(model, xtrain, ytrain, xval, yval, n_epoch, batch_size)    print(“— — — — — "*5)

Since the model is evaluated, we can calculate the average of the accuracy and loss, which will be shown in the result section.

5. Export CNN model

To export the model, we need to alter the code in that we have split the dataset into a testing set and a training set before the training process. The model will not be aware of the test set until the final evaluation process. In contrast, the validation set is used to tune the model in the training process.

from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=42)print(“y_train: number of samples each class: {}”.format(np.unique(y_train, return_counts=True)))print(“y_test: number of samples each class: {}”.format(np.unique(y_test, return_counts=True)))

In the process, we include a callback named ModelCheckpoint. It helps us save the best performer when val_accuracy has improved.

cnn_model = None
cnn_model = create_model(image_shape=(256, 4096, 1))# Define the EarlyStopping function
earlystopping = EarlyStopping(monitor="val_accuracy", patience=2)# Define the path for ModelCheckpoint function
filepath = project_path + “/model/weights_best.hdf5”# Define the ModelCheckpoint function
checkpoint = ModelCheckpoint(filepath, monitor="val_accuracy", verbose=1, save_best_only=True, mode="max")callbacks_list = [checkpoint, earlystopping]# Train model
cnn_model.fit(X_train, y_train, batch_size=10, epochs=10, validation_split=0.2, callbacks=callbacks_list)

We exported the model weights into a file weights_best.hdf5. The requirements for model reconstruction are:

  1. Model. The original model from create_model()
  2. Weights. Neural connection weights which produced after being trained by the dataset

(Optional) Import model into the edge device

Fig 5: Raspberry PI with a camera. Source

The application can be used in the manufacturing process to detect fabric defects by using our model based on the Edge AI concept. Although we do not have the devices, this article can guideline the procedure, which is not a complex implementation. Let us assume that a Raspberry PI connected with a camera or image feed is provided. Since we need to use the same CNN model, the prerequisite program of Raspberry PI should be the same as our article (e.g. Python, TensorFlow, etc.), as you can read in How-to-install-TensorFlow-on-Raspberry-Pi. Also, dependencies for using camera devices acquire to install such as PiCamera, OpenCV, etc. Read here: Accessing-the-Raspberry-Pi-Camera-with-OpenCV-and-Python. For the predictive program, it should consist of create_model(), model.fit(), and parts of reading the input image from the camera.

For the example code:

https://gist.github.com/ad9eef4b24a30c7aa1ea8c5d0eaec57d

# DefectDetectionPrdictive.py# Import the necessary dependency
from tensorflow.keras import datasets, layers, models, optimizers, regularizers
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from picamera.array import PiRGBArray
from picamera import PiCamera...# Initial create CNN, Same function as 3.
model = create_model()
# Apply weights to CNN
model.load_weights("weights_best.hdf5")
# Initialize PiCamera
camera = PiCamera()
# Capture image from camera
rawCapture = PiRGBArray(camera)...# Grab an image from the camera
camera.capture(rawCapture, format="gray")
# Predict the input image
print(model.predict(image))...

To compile the program:

python3 DefectDetectionPrdictive.py

Note — This part is not an actual demonstration. Therefore, errors may occur. This is simply the concept of such implementation. Furthermore, the program can be adjusted depending on the proposed system specification. This idea of its application in edge device implementation for the smart factory has been discussed extensively in various research. For instance, D. Huang et al., 2018 presented the architecture of the smart factory in the defect detection systems.

Result

The graphs below illustrate the performance of the model between each training epoch.

Our model accuracy:

Higher is better

The maximum epoch count is ten iterations. But due to the early stopping threshold, the model would halt learning when there is no significant improvement between epoch, hence 3 to 7 epochs on the performance graph.

Our model loss:

Lower is better

We select the best epoch for each fold to calculate the validation average:

average accuracy: 0.5862, average loss: 0.968

All in all, the result is within satisfaction given the less than ideal size of the datasets, and the demonstration for the implementation of this concept is still clear.

The exported model performance:

model = create_model()
model.load_weights(project_path + “/model/weights_best.hdf5”)
score, acc = model.evaluate(X_test, y_test, verbose=0)
print(score, acc)

In which the result is:

(0.7061388492584229, 0.625)

This is the model evaluation by test set where “score” represents the model loss, and “acc” means the model accuracy.

Conclusion

In this article, we have demonstrated and go over many steps regarding CNN based implementation for edge-based defect detection, including:

  • Fabric Defect Detection using CNN implementation
  • The defect image dataset from J. Silvestre-Blanes et al., 2019
  • AlexNet model Infrastructure
  • Stratified K-fold model evaluation
  • Our model has an average accuracy of 0.5862 and an average loss of 0.968
  • Edge Device Implementation

While the model performance is as expected given the lack of a sizable dataset, a major problem that we can observe is overfitting, which can be remedied in many ways such as:

  1. Increase the dataset — Collect more data, data synthesis, or data augmentation.
  2. Change the architecture — Try other CNN architecture such as VGG-16 or ResNet-50.
  3. Tuning model — Better architecture fine-tuning can cause the model to achieve better performance.
  4. Transfer Learning — This is a method of deploying a pre-trained model and later train additionally using our dataset by freeze the pre-trained layers.

In conclusion, while the methodology in this article can use many improvements, we hope that this demonstration is a positive contribution as a stepping stone to the growing popularity of edge-based AI and that you, the readers, may find it useful.

GitHub

The source code with comments is provided. Please check:

sadanalog/defect-detection-using-cnn

This project is a part of ICT730 Hardware Designs for Embedded Systems in the TAIST-Tokyo Tech program which has…

github.com

Acknowledgment

This report is a part of ICT730 Hardware Designs for Embedded Systems in the TAIST-Tokyo Tech program, which has been written by Isada Sukprapa and Naruson Srivaro.

Reference

Defect Detection
Edge Ai
Deep Learning
Convolutional Network
Smart Factory

--

--

Isada Sukprapa
Isada Sukprapa

Written by Isada Sukprapa

28 Followers
3 Following

Python Software Engineer

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams