Essays, Posts & Presentations

Using Neural Networks to Boost Student Learning in Chemistry

Arun Sharma

Author

Arun Sharma

Title

Using Neural Networks to Boost Student Learning in Chemistry

Description

Professor's innovative Introduction to Scientific Computing course equips students to apply computational thinking to solve scientific problems. Chemistry-based set of examples utilizes Mathematica and the Wolfram Language.

Using Neural Networks to Boost Student Learning in Chemistry

July 27, 2021

Arun Sharma, Guest Author

I attended the Wolfram Neural Networks Boot Camp 2020, and that inspired me to incorporate elements of data science and machine learning in my course. The helper functions for machine learning make it quite easy to experiment and introduce such applications to students. We chose to perform image recognition and classification problems that are routinely used to initiate the topics of both neural networks and machine learning.

I was immediately struck, however, by the lack of chemistry-oriented image datasets, and after playing with a Modified National Institute of Standards and Technology (MNIST) handwriting dataset example, decided to create this type of dataset. The goal was to provide students with an end-to-end data science project experience, from creating the data to the final step of performing an advanced machine learning exercise. Thus, this project creates a set of images of glassware routinely found in a chemistry laboratory with the goal of constructing our own example of object identification.

Methodology The following steps were outlined to collect pictures for each piece of glassware:

1. Take pictures of glassware when empty and when filled with water.

2. Fill the glassware with different-colored solutions to various levels, e.g. a 250-mL Erlenmeyer flask would be filled to various volume capacities.

3. Provide variation in the background by occasionally including clutter. For example, some glassware was placed on the laboratory bench or an elevated surface to provide different backgrounds.

4. Organize all pictures in folders shared on Google Drive. Students uploaded pictures from their phones to the shared drive.

The item classes—and the number of items in each class—for the classification problem are:

Beaker	30
Büchnerfunnel	20
Burette	11
Burettestand	5
Erlenmeyerflask	24
Flat-bottomflask	18
Funnel	23
Graduatedcylinder	47
Pipette	16
Round-bottomflask	24
Separatoryfunnel	22
Standardmeasuringflask	36
Testtube	6
Test-tubecollection	17
Test-tubestand	14
Viscometer	14
WashBottle	8
Total	335

This collage has one picture of each of the 17 different types of laboratory equipment. There were some difficulties with the pipet pictures. The 5-mL pipets appeared quite difficult to detect in the picture, so we used white or yellow paper backgrounds to highlight them. Of course, there are some other issues with labeling. Some glassware is routinely seen and utilized as suspended, e.g. burettes and separatory funnels.

Volumetric glassware that could be filled with liquids had far more pictures than other items in the dataset. This collection of beaker images illustrates the diversity:

The important point here is that the data collection included multiple variations: lighting conditions, focus on the object, cellphone quality, picture presets and various images for each label.

Dataset Augmentation Due to the small number of images in the dataset and the disparity in the number of images between different classes, we thought it prudent to increase the sample space by performing some image modifications on the pictures. We utilized

ImageEffect

options to increase the sample size by a factor of 19:

In[]:=

imageSetAugmentation[objectImages_List]:=Module[{detailEnhanced,blurredImages,noisyImages,lightDarkImages,reflectedImages,rotatedImages,augmentedImages},detailEnhanced=ImageEffect[#,"DetailEnhancing"]&/@objectImages;blurredImages=Blur[#,RandomInteger[{1,3}]]&/@objectImages;noisyImages=ImageEffect[#,"Noise"]&/@objectImages;lightDarkImages=Join[Lighter[#]&/@objectImages,Darker[#]&/@objectImages];reflectedImages=ImageReflect[#,Left]&/@Join[objectImages,detailEnhanced,blurredImages,noisyImages,lightDarkImages];rotatedImages=ImageRotate[#,RandomInteger[{-10,10}]Degree]&/@Join[objectImages,detailEnhanced,blurredImages,noisyImages,lightDarkImages,reflectedImages];augmentedImages=Join[objectImages,reflectedImages,rotatedImages];Return[augmentedImages];]

Images were blurred using the

Blur

function with a randomly chosen pixel radius over which the blur was applied. Images were also modified to appear lighter or darker using the appropriately named functions. Next, all of these images were collected and flipped from left to right. The final operation randomly rotated all images between –10 and 10 degrees.

Here is the result of these operations on a single picture from the Erlenmeyer flask label:

The effect of this series of operations on a single beaker-class image is shown here:

In[]:=

ListAnimateimageSetAugmentation

,AnimationRunningTrue,AnimationRepetitions3,PaneledFalse

Out[]=

Now we’ll represent the effects of image augmentation for one image chosen from each of the beaker, flat-bottom flask and wash-bottle classes, respectively:

In[]:=

ListAnimateimageSetAugmentation

,AnimationRunningTrue,AnimationRepetitions3,PaneledFalse

Out[]=

Image set augmentation produced 6,365 images. We then decided to use this as our dataset for training and testing of various pre-trained neural networks available at the Wolfram Neural Net Repository.

Neural Networks We decided to evaluate the performance of four well-known pre-trained neural networks from the field of image recognition. Here is a summary of the sources, number of layers and parameters of these neural networks:

Network	Year	Source	Layers	Parameters
Inceptionv1	2014	Google	147	6,998,552
Inceptionv3	2015	Google	311	23,885,392
ResNet–50	2015	Microsoft	177	25,610,216
ResNet–101	2015	Microsoft	347	44,654,504

Data Import and Organization We split the dataset to use 80% of the data as the training set and the remaining 20% as the testing set. We imported the training and testing sets and carried out the training and evaluation of classification performance using four variants:

1. Full-color images

2. Full-color images with the dataset augmented using the image augmentation module described earlier

3. Grayscale images from the full-color images

4. Grayscale images with the dataset augmented using the image augmentation module described earlier

Let’s check some examples of the training and testing sets:

In[]:=

RandomSample[trainingSet,3]

Out[]=



Grad. Cylinder,

Buchner Funnel

In[]:=

RandomSample[testingSet,3]

Out[]=



Std. Flask,

Viscometer,

Separatory Funnel

After the 80–20 split, here are the total images in the training and test sets:

Trainingsetimages	5092
Testingsetimages	1273
Total	6365

Training and Results The following steps were used for net surgery on the pre-trained ResNet model obtained from the Wolfram Neural Net Repository:

1. Remove the linear layer from the pre-trained net.

2. Remove the final classification layers of the trained net and replace them with this classification task:

In[]:=

net=NetModel["ResNet-50 Trained on ImageNet Competition Data"]

Out[]=

NetChain



Input port:	image
Output port:	class



In this example, the

LinearLayer

and

SoftmaxLayer

functions have been removed, and a new net composed of the pre-trained net followed by a linear layer and a softmax layer is created:

In[]:=

tempNet=NetDrop[net,-2]

Out[]=

NetChain



Input port:	image
Output port:	vector (size: 2048)



In the final classifier step, LinearLayer takes an integer argument for the number of classes. The number of classes for the laboratory glassware identification experiment is the following:

In[]:=

classLabels={"Beaker","Buchner Funnel","Burette","Burette Stand","Erlenmeyer Flask Chem","Flat Bottom Flask","Funnel","Grad. Cylinder","Pipette","Round Bottom Flask","Separatory Funnel","Std. Flask","Test Tubes","Test Tube","Test Tube Stand","Viscometer","Wash Bottle"};

In[]:=

numClasses=Length[classLabels];(*17classesfromthesectionondataimportandorganization*)

An uninitialized LinearLayer and a SoftmaxLayer is attached to create the final net. A

NetDecoder

with the labels specified by the laboratory glassware is attached to inference the class names from the output of the net:

In[]:=

newNet=NetAppend[tempNet,{"classifier"LinearLayer[numClasses],"probabilities"SoftmaxLayer[]},"Output"NetDecoder[{"Class",classLabels}]]

Out[]=

NetChain



uniniti

aliz

Input port:	image
Output port:	class



Here we will perform training with

MaxTrainingRounds

set to 5. The training is finished in about five minutes on a laptop with an NVIDIA Quadro M2000M GPU. For the transfer learning, all the layers except the final classifier layer are kept frozen:

In[]:=

trainedNet=NetTrain[newNet,trainingSet,All,ValidationSetScaled[0.1],LearningRateMultipliers{"classifier"1,_0},MaxTrainingRounds5,TargetDevice{"GPU",All}]

Out[]=

NetTrainResultsObject

NetTrain Results

summary

batches:

360,

rounds:

time:

5.2min

examples/s:

74.

data

training examples:

4608,

validation examples:

512,

processed examples:

23040,

skipped examples:

method

ADAM

optimizer

batch size

64,

GPU

All

round

loss:

3.39×

-2

error:

0.109%

validation

loss:

4.46×

-2

error:

0.586%

training set

validation set



Various properties can be obtained from the trained net. Here we list a few:

In[]:=

trainedNet["TotalTrainingTime"]

Out[]=

313.258

In[]:=

trainedNet["FinalPlots"]

Out[]=

Loss

	rounds
loss	Contents cannot be rendered at this time; please try again later

	validation
	training

,ErrorRate

	rounds
error rate	Contents cannot be rendered at this time; please try again later

	validation
	training



Classification Performance

The trained net can be directly used with the

ClassifierMeasurements

function to obtain a varied range of performance metrics:

In[]:=

classifier=ClassifierMeasurements[trainedNet["TrainedNet"],testingSet]

Out[]=

Classifier Measurements

Classifier method	Net
Number of test examples	1273
Accuracy	(99.53 ± 0.19 ) %
Accuracy baseline	(14.1 ± 1.0 ) %
Geometric mean of probabilities	0.952 ± 0.0037
Mean cross entropy	0.0497 ± 0.0039
Single evaluation time	110. ms/example
Batch evaluation speed	11.3 examples/s
Contents cannot be rendered at this time; please try again later

Obtain the overall accuracy of the trained net on the test set:

In[]:=

classifier["Accuracy"]

Out[]=

0.995287

"ConfusionMatrixPlot" is a great plotting technique to visualize the correctness of the classification. The off-diagonal terms are the incorrectly classified examples:

In[]:=

classifier["ConfusionMatrixPlot"]

Out[]=

Contents cannot be rendered at this time; please try again later

Here are the top five of the six errors where the net misclassified:

In[]:=

classifier["TopConfusions"5]

Out[]=

{Flat Bottom FlaskStd. Flask,FunnelStd. Flask,BeakerBuchner Funnel,BeakerBurette,BeakerBurette Stand}

The low frequency of misclassification is clear from the confusion matrix. A graduated cylinder gets misclassified as a standard flask only once out of 179 samples tested. The high accuracy with very little computational cost is quite a revelation for students. Another striking example is the Büchner funnel classification with perfect precision and recall. The test-tube class, which has the lowest number of images in the test set (23), also displays perfect recall:

Results Overview Following the processes described previously, we tested the four neural networks on four variants of our laboratory images.

The validation error during the training is initially quite high with the unaugmented dataset. After a few rounds of training, both datasets resulted in trained networks with increased accuracy:

Error Fraction Augmented Set

Error Fraction No Augmentation

Contents cannot be rendered at this time; please try again later

The next graphic shows the training time and accuracy of all four trained networks with each of the four dataset variants. Unsurprisingly, the enlarged datasets take longer to train and provide greater accuracy compared to the unaugmented datasets. The ResNet-50 and ResNet-101 networks both provide similar classification accuracy between grayscale and full-color images:

Augmented dataset accuracy

Unaugmented dataset accuracy

Contents cannot be rendered at this time; please try again later

Teaching Ideas and Conclusion The dataset images and sample notebooks used for the training of networks and data analysis are available on Zenodo. This should enable other instructors to employ this project in its entirety or devise other projects using its data. One simple extension would be to add different glassware images and investigate classification performance. Another interesting, and possibly more advanced, application would be to identify the text on glassware that annotates the volume, especially on beakers or Erlenmeyer flasks. This project also provides a hands-on experience about the power of deep learning and pre-trained neural networks for students from non–computer science backgrounds.

Acknowledgements I am grateful to my students in the Introduction to Scientific Computing course at Wagner College. I am also thankful to Dr. Tuseeta Banerjee and Dr. Mads Bahrami for the invitation to write this blog post and the Wolfram Blog Team for their support in publishing it.

Arun Sharma is an associate professor of chemistry at Wagner College in New York City. His research and teaching interests include computational chemistry, machine learning and the infusion of computing into all levels of the undergraduate chemistry experience. He leads a research group of undergraduate students on a variety of projects that include molecular dynamics simulations, quantum chemistry calculations and applications of machine learning to chemical problems. The National Science Foundation and the Petroleum Research Fund of the American Chemical Society have supported his research group.

Relevant Publications

[1] A. K. Sharma, “Laboratory Glassware Identification: Supervised Machine Learning Example for Science Students,” Journal of Computational Science Education, 12(1), 2021 pp. 8–15. doi.org/10.22369/issn.2153-4136/12/1/2.

[2] A. K. Sharma, “Glassware Images and Code Samples for Training and Identification of Glassware by Neural Networks,” Zenodo (May 6, 2021). doi.org/10.5281/zenodo.4019356.

Cite this as: Arun Sharma, "Using Neural Networks to Boost Student Learning in Chemistry" from the Notebook Archive (2021), https://notebookarchive.org/2021-08-7vk893s