Keeping an "eye" on the experiment: computer vision for real-time monitoring and control

Abstract

This work presents a generalizable computer vision (CV) and machine learning model that is used for automated real-time monitoring and control of a diverse array of workup processes. Our system simultaneously monitors multiple physical outputs (e.g., liquid level, homogeneity, turbidity, solid, residue, and color), offering a method for rapid data acquisition and deeper analysis from multiple visual cues. We demonstrate a single platform (consisting of CV, machine learning, real-time monitoring techniques, and flexible hardware) to monitor and control vision-based experimental techniques, including solvent exchange distillation, antisolvent crystallization, evaporative crystallization, cooling crystallization, solid-liquid mixing, and liquid-liquid extraction. Both qualitative (video capturing) and quantitative data (visual outputs measurement) were obtained which provided a method for data cross-validation. Our CV model's ease of use, generalizability, and non-invasiveness make it an appealing complementary option to in situ and real-time analytical monitoring tools and mathematical modeling. Additionally, our platform is integrated with Mettler-Toledo's iControl software, which acts as a centralized system for real-time data collection, visualization, and storage. With consistent data representation and infrastructure, we were able to efficiently transfer the technology and reproduce results between different labs. This ability to easily monitor and respond to the dynamic situational changes of the experiments is pivotal to enabling future flexible automation workflows.

Fig. 1. (a) Universal vision-based outputs in workups monitored by HeinSight2.0; (b) overview of interrelated components of HeinSight2.0: classification outputs (from CNN), quantification outputs (from image analysis), and process variables from iControl; (c) integrations and applications of HeinSight2.0 CV system.

Fig. 2. Experimental workflow setup of HeinSight 2.0, illustrating the integration of EasyMax hardware components (dosing unit, overhead stirrer, temperature probe, and reaction vessel), webcam (within an enclosure), and software components ((CV model and iControl) with real-time trend visualization).

Fig. 3. (a) Illustration of the experimental setup of solvent exchange distillation. (b) Solvent exchange distillation for a reaction slurry containing a mixture of EDCI–HCl, and triethylamine in acetone, charging 10 mL of butanone when the threshold volume is reached. (c) Constant volume antisolvent crystallization via solvent exchange distillation of acetaminophen in MeCN. For both (b) and (c), the internal temperature of the reactor (red) was captured via the EasyMax temperature probe, while the reactor volume (purple) was returned from the CV system. The apparent spread in volume recorded by the CV system was due to variations in the liquid height due to fluctuation from mechanical agitation and boiling, which both changed the 2D cross sections presented to the camera; see ESI for detailed measurement and control workflow. The volume trend data were visually emphasized by adding manual markers to guide the eye.

Fig. 4. (a) Illustration of the experimental setup for evaporative crystallization of acetaminophen in MeCN. (b) Monitoring evaporation of MeCN over time to observe acetaminophen nucleation. The reactor volume (purple), turbidity (yellow), and solid (pink) were captured by the CV system. (c) Illustration of the experimental setup for cooling crystallization case study. (d) Feedback loop using real-time data to determine solubility and MSZW of acetaminophen in MeCN. The internal temperature of the reactor (red) was captured via the EasyMax temperature probe, while the turbidity (orange) and homogeneity (purple and yellow) were returned from the HeinSight2.0 model.

Fig. 5. (a) Illustration of the experimental setup for agitation case studies. (b) Monitoring the presence of solids to determine the minimum stirring speed for effective agitation of a slurry of acetaminophen in MeCN. The stir rate of the reactor (grey) was captured via the EasyMax overhead stirrer, while the turbidity (orange) and solid (pink) were returned from the CV system. (c) Monitoring turbidity to determine the settling time for a slurry of acetaminophen in MeCN.

Fig. 6. (a) Illustration of settling kinetic behavior of the separation of two immiscible solutions. (b) Monitoring liquid level to determine the separation time of a mixture of DCM and water. The stir rate of the reactor (grey) was captured via the EasyMax overhead stirrer, while the volumes 1 and 2 (purple and pink) were returned from the CV system. (c) Scheme of (–)TBZ and (–)CSA sodium salt separation in DCM and water. (d) Monitoring the liquid level to determine the separation time of a mixture of (–)TBZ and (–)CSA sodium salt in DCM and water.

Fig. 7. Conceptual illustration of how multi-outputs monitoring enables modular automation for diverse workflows.