Some Information About the Data Source

The raw data we will use to better understand Kawasaki disease1 comes from a database hosted on the NCBI (National Center for Biotechnology Information).

The GEO database (Gene Expression Omnibus) is a repository that contains various biological samples for which high-throughput gene expression analyses have been performed, i.e., measurements of gene activity. Analytical techniques such as microarrays or genomic chips enable large-scale screening, detection, and quantification of these variations.

Initial Data Format

The raw data is presented as a data table of approximately 31MB with the following characteristics:

• 135 columns representing samples taken from patients diagnosed with Kawasaki disease (KD) or belonging to a control group.

• 60,075 rows representing distinct genetic sequences, primarily genes. Each gene is referenced by a unique ID ("ensembl_gene_id").

• Over 8 million values, corresponding to the quantified amount of mRNA for each gene in different patients. The unit of measurement is TPM (Transcript Per Million), a standardized metric used to quantify gene expression. This value reflects gene expression levels, as messenger RNA acts as the intermediate vector between genes and proteins.

Some Preliminary Data Cleaning Steps

To process this dataset, a few relatively simple actions are necessary:

1. The "genename" column is removed because its information does not always align with official nomenclature.

2. The data matrix is transposed, meaning rows and columns are swapped.

3. Numeric values are rounded, significantly reducing computational power requirements. Since only the magnitude matters, precise decimals are unnecessary.

The resulting dataset is accompanied by a metadata table indicating whether patients are healthy or affected by the disease.

Processing Data with the DESeq2 Package

DESeq22 is a bioinformatics tool used to analyze differences in gene activity between two or more populations. It helps identify which genes exhibit significant changes in expression and are more or less active under different biological conditions.

DESeq2 integrates the dataset, normalizes it for comparability, and applies a variety of tests to detect significant differences between the studied populations. This process is known as Differential Expression Analysis (DEA)3.

Without delving into all the technical details, DESeq2 uniformly applies numerous statistical hypothesis tests to compare the expression of each gene between healthy and diseased patients. For each test, a p-value is generated, indicating whether the result is statistically significant. The strength of this package lies in its statistical assumptions tailored to genetic data (e.g., negative binomial distribution) and its automated corrective measures to ensure robust and comparable results.

Here’s what we obtain after running DESeq2:

While the results may seem complex, there are two key values to focus on:

1. Adjusted p-value (padj): Indicates whether the difference between healthy and diseased patients is statistically significant. This helps rank genes to highlight the most differentiating ones.

2. Log2FoldChange: Shows whether a gene is overexpressed (positive values) or underexpressed (negative values) in diseased patients.

With this information, we have all the necessary tools to begin a differential analysis of Kawasaki disease.

The OMIX Buddy Application

Alongside this article, you can:

• Visualize and interact with the graphs presented here using the web application I created for the occasion: The OMIX buddy 🤖 !

• Access the complete codebase as a notebook4, for those who want to follow the detailed steps.

First Level of Analysis: At the Gene Level

When studying the impact of a disease on gene expression, two complementary approaches can be taken:

1. Individual gene analysis, which identifies specific genes whose expression changes significantly.

2. Functional family analysis, which examines how these changes cluster into broader biological categories.

Individual Analysis: The Volcano Plot

The Volcano plot5 is a commonly used representation in bioinformatics for Differential Expression Analyses.

A volcano plot is a scatterplot that displays:

• On the x-axis: The Log2FoldChange value. Genes on the right are overexpressed in the diseased group compared to the control group, while genes on the left are underexpressed.

• On the y-axis: The adjusted p-value. Typically, the negative logarithm of the p-value is used to better distribute the data and enhance readability. Genes higher on this axis show greater expression differences between healthy and diseased patients.

The genes of greatest interest are represented within the two oval regions at the top left and right of the plot. Generally, thresholds are defined on both axes to filter the population and retain only the most relevant data points, which is why some points appear grayed out in the visuals.

For Kawasaki disease, I selected 8 genes of interest for further analysis. These are promising candidates as potential biomarkers of the disease.

Functional Family Analysis

This second approach leverages a complementary database called GO (Gene Ontology)6. Researchers and biomedical scientists contribute to public databases that catalog genes, their families, and their functions, collectively known as ontologies.

The Gene Ontology database organizes genetic information into three key dimensions:

1. Biological processes: The phenomena genes are involved in (e.g., inflammatory response or cell division).

2. Molecular functions: The specific activities of proteins encoded by these genes (e.g., binding to DNA or catalyzing chemical reactions).

3. Cellular components: The cellular structures or compartments where the genes are active (e.g., cell membrane or mitochondria).

By cross-referencing the previous dataset with this new source of information, we can identify enriched functional categories and associated genes.

In the case of Kawasaki disease, it appears that the gene families primarily involved are those associated with protein binding and RNA binding. Underexpression is predominantly observed in the cytosol and nucleoplasm.

Conclusion

In this article, we explored the power of omics data analysis, transforming over 8 million raw data points into two clear and insightful visualizations. This approach not only enhances our understanding of Kawasaki disease but also paves the way for promising therapeutic avenues.

In the next article, we will take the analysis one step further by focusing on the patient level. We will explore how data analysis can revolutionize clinical trial design by identifying more homogeneous patient groups, paving the way for truly personalized medicine.

… As usual … Let’s rock with data ! 🧬🤘

I must admit right away that this educational project is a bit of a challenge for me, as biology is a science with its own concepts and jargon. I will therefore make an extra effort to simplify and clarify. I encourage you to stick with me, even if you're unfamiliar with the pharmaceutical R&D sector, as I genuinely believe the tools we will explore are worth knowing.

The Challenges of Modern Medicine

Pharmaceutical R&D faces several significant challenges:

• Around 90% of drug candidates never make it to market as approved medicines.1

• The average ROI for pharmaceutical R&D has dropped from 10% in 2010 to 1.8% in 20202.

• Development costs are increasing exponentially, with the failure of a molecule in Phase III R&D potentially costing up to $1 billion.

Moreover, identifying new active molecules to treat increasingly complex diseases has become more challenging. Far from the "simple" diseases caused by a predominant factor, most conditions that resist modern medicine are multifactorial (e.g., cancers, autoimmune diseases, neurodegenerative disorders, to name a few).

To identify new active molecules, it is therefore imperative to better understand the biological mechanisms underlying these pathologies to pinpoint relevant therapeutic targets. However, these conditions often involve hundreds of genes, thousands of proteins, and interacting environmental factors.

How can we identify the root causes of a disease when thousands of parameters are at play? How can we develop effective treatments when each patient reacts differently?

Guess what? … Data Science is a powerful tool for the job!

Omics Data: A 360° View of Life

What is it?

The term “-omics” refers to various fields of biology that all end with the suffix "-omics." The four main components are as follows3:

1. Genomics: The study of our genes, which define our genetic heritage. The data involved consists of DNA sequences.

2. Transcriptomics: The study of gene expression and activity. The data involved consists of mRNA sequences (the same fragments found in some modern vaccines).

3. Proteomics: The study of proteins, the true workers of our cells. The data includes amino acid sequences that make up proteins, as well as the 3D structure of these proteins.

4. Metabolomics: The study of biochemical processes and metabolic cascades occurring in our bodies.

Specific Data Science Challenges in R&D

I could mention many points, but I have limited myself to three main challenges:

1. The Overwhelming Volume of Data
The average size of a human genome is about 3.2 billion base pairs, equivalent to approximately 200 GB of raw data (about 50 HD movies!)4.

...and as we saw earlier, this is just the first layer of data—the genome. Imagine the sheer volume of omics data associated with just one patient!

Even a simple bacterium like E. coli has 4,300 genes, amounting to a sequence of about 4.6 million nucleotides.

2. The Unique Structure of Omics Data
Unlike traditional datasets, omics datasets are often “horizontal”: they have a limited number of samples or patients but an extreme number of variables. This is sometimes called “fat data”5 as these datasets are rich with numerous, often redundant, variables.

Specific tools to reduce dataset dimensionality or select relevant variables are necessary to extract insights effectively and efficiently.

3. Specialized Domain Language
As mentioned in the introduction, working with omics data requires effort to understand:

• Biological jargon and concepts (e.g., co-expression networks, Differential Expression Analysis).

• Technical jargon related to data often stored in formats and standards unfamiliar to the general public6.

Why Are Omics Data Essential ?

In the drug development cycle, data from biological profiling plays a crucial role, offering significant advantages at every stage of the process. These benefits are clearly illustrated in the infographic below7.

Biological data are useful:

1. At the disease level, in basic research, for:

• Understanding pathological mechanisms: Omics data deciphers the complex biological processes involved in disease development.

• Identifying therapeutic targets: It helps discover specific genes, proteins, or biological pathways that could be targeted by new therapies.

2. At the patient level, in clinical research, for:

• Precise patient stratification: By segmenting populations based on their biological characteristics, clinical trials can recruit better-suited participants, reducing trial duration and cost while increasing the chances of success.

• Optimized clinical study monitoring through predictive biomarkers: These markers enable earlier and more accurate evaluation of the effectiveness of tested treatments.

Omics data is not limited to basic and clinical research. It also has applications in related fields such as cell culture bioprocesses, pharmacovigilance, and personalized medicine.

Our Practical Application: Exploring Kawasaki Disease

Kawasaki disease8 is a rare pediatric illness that causes inflammation of blood vessels and can lead to severe cardiac complications. This disease recently gained attention due to an increase in prevalence among children during the COVID-19 pandemic.

The exact cause of this disease remains poorly understood.

I chose this condition to walk you through an analysis of biological data from patients affected by Kawasaki disease. Anonymized data is available online via the National Library of Medicine 9.

Our project will focus on creating a graphical interface to explore different genes associated with Kawasaki disease to identify potential biomarkers of interest.

A biomarker is a measurable characteristic that indicates normal biological processes, disease processes, or responses to interventions, with qualified biomarkers aiding drug development by reducing uncertainty in regulatory decisions within a defined context of use. - FDA10

Generally, I always try to convey the concept of return on investment (ROI) for each project at “The Pharm’AI Company.” In this case, I won’t be able to do so, as calculating ROI in R&D is quite complex. Indeed, identifying biomarkers helps reduce the risk of project failure, improve patient selection, and/or shorten project duration. However, all of this is very difficult to translate into hard cash.

Conclusion

With the basics laid out, in the next article, I will introduce “The Omix Buddy” my application, which will allow us to explore the dataset related to Kawasaki disease.

I will present visualizations that help researchers quickly identify genes of interest and understand their interactions, transforming mountains of data into biological insights.

En attendant … Let’s rock with Biology & Data ! 🧬

The application prototype is available here.

Redefining the Problem

Let’s recap some key points from the second article to quickly recontextualize the problem from a “Data” perspective.

We monitored a hundred turbines over an extended period. This monitoring allowed us to collect numerical readings from over twenty sensors (temperature, pressure, vibrations, etc.), as well as an assessment of failure risk through the TTF (Time-To-Failure).

The goal of our model is to predict the risk level of failure based on new input data, assigning it to one of the following three categories:

• Class 0: Normal operation (no risk)

• Class 1: Caution zone (moderate risk, maintenance recommended)

• Class 2: Critical zone (high risk of imminent failure)

In terms of modeling, the task is to predict one of three categories. In Data Science, this is referred to as a multi-class classification problem.

From an operational perspective, the most critical category for us is Class 1, as this is the ideal moment to trigger maintenance actions.

Choosing the Right Algorithm

It’s often challenging to determine in advance which algorithm will best suit a given problem. There’s no universal algorithm that outperforms all others in every scenario. In Machine Learning, this reality is well encapsulated by the theorem formulated by David Wolpert, famously known as the “No Free Lunch” theorem1.

There is no free lunch! — David Wolpert

This principle underscores that no algorithm is optimal for every problem; each tool has its strengths and weaknesses depending on the context.

During the development phase, it’s common to evaluate multiple algorithms in parallel to choose the one offering the best trade-offs based on the project’s criteria. These criteria include prediction quality, model explainability, inference time, and even energy consumption.

To simplify this project and avoid unnecessary complexity, I opted not to compare a range of algorithms but instead to focus directly on a specific method, i.e. Random Forest2.

This algorithm typically provides a good balance between performance and computational efficiency.

For a detailed explanation of how decision-tree-based algorithms work—particularly Random Forest—please refer to the link and video3 below.

Training the Model

Training a Machine Learning algorithm involves several strategic steps to ensure the reliability of our predictions. Below is a summary of these steps, but the full code is, of course, available in this notebook4.

Step #1: Data Preparation

Our initial dataset shows a marked imbalance: 17,531 observations correspond to normal operation, while only around 1,500 observations each represent moderate and high-risk states. This is quite logical—machines are more often in good working condition than close to failure.

To address this imbalance, which could bias our model, I used a statistical technique called SMOTE, which generates synthetic examples to balance the classes. Sensor measurements were also standardized (NB: Standardization is unnecessary for tree-based approaches like Random Forest but is essential for methods like PCA).

Step #2: Creating a Performance Metric

By definition, an algorithm will make prediction errors; a 100% accurate model doesn’t exist. For this project, I wanted to minimize prediction errors in Class 1 at all costs, as this category triggers maintenance actions.

The choice of performance metric is crucial because it directly influences and guides the training phase. I developed a custom performance metric (recall * 0.7 + accuracy * 0.3) that penalizes misclassifications in Class 1 more heavily.

This strategy does lead to some false positives, resulting in slightly excessive maintenance actions. However, this approach aligns with the project’s priority: avoiding missed opportunities for preventive maintenance.

Step #3: Training the Model

As with all algorithms, the model training phase involves various optimization processes to converge toward the optimal parameters for the Random Forest model.

A brief technical explanation for Data Science enthusiasts (apologies to the uninitiated 🤯):

Hyperparameter optimization was conducted using GridSearchCV, which tests different combinations of parameters, including:

• Tree depth (from 4 to 8 levels),

• Splitting criteria (Gini or entropy),

• Minimum number of samples per split.

A classic train/test split approach combined with 5-fold cross-validation and random permutations ensures robust results.

Step #4: Performance Evaluation

The results are promising: our model achieves 87% recall for detecting turbines in Class 1 and, most importantly, misses no critical Class 2 cases. Although the model can be overly cautious, misclassifying compliant turbines as risky, this trade-off is acceptable in a safety-first context.

Detection isn’t perfect, but that’s not a major concern. Class 1 spans about twenty cycles, making it nearly impossible to overlook a machine entering the red zone.

The only potential risks are encountering a new failure mode not captured during training or a sudden, unexpected breakdown.

Conclusion

An analysis of the most influential sensors reveals the critical importance of:

• Sensors s7 and s11 (measuring compressor discharge pressure),

• Sensor s4 (monitoring the low-pressure turbine outlet temperature),

• Sensor s12 (tracking the fuel flow ratio).

These sensors play a key role in our model’s predictions. Beyond modeling, it would be beneficial to incorporate these measurements into reference documentation and training programs as “sentinels” for monitoring equipment health.

From Prototype to Production

Solution Architecture

Transitioning from a prototype to an operational solution often represents a major challenge in data science projects. To make the model accessible and usable, I opted for a simple yet effective architecture based on an interactive web application developed with Streamlit5 (NB: I’ve previously discussed this framework in earlier projects).

Webapp overview

The user interface developed with Streamlit includes the following features:

1. Simulating new sensor data:

   A button generates random new configurations of values outside the training dataset.

2. 3D Visualization of Results:

   A PCA-based visualization shows the machine’s new state relative to historical data.

3. Prediction Panel:

   For each new configuration, the app displays the probabilities for each risk level, along with clear maintenance action recommendations.

Potential Improvements

Real-Time Integration

The data used for this project was batch-collected. However, real-time sensor data is becoming the standard for such applications. This would involve:

• Ingesting continuous data streams.

• Preprocessing signals to reduce noise in input data.

• Using Deep Learning algorithms capable of detecting anomalies in time series, coupled with an automatic alert system.

This predictive maintenance project clearly calls for a follow-up project with continuous data. (NB: This is on my to-do list 😃!)

Enriching the Dataset

The tool we’ve created is purely mathematical, based solely on sensor data. However, we could build more sophisticated tools by combining raw data with:

• CMMS data (Computerized Maintenance Management System—detailed intervention history),

• Functional analyses of equipment,

• Incident reports and failure descriptions,

• Maintenance technicians’ and engineers’ expertise,
  ...and more.

This would enable predictions not only of failure risk but also failure modes, root causes, and priority corrective actions, resulting in a much smarter tool capable of guiding the type and content of predictive maintenance interventions.

Advanced Monitoring System

The model was built using data from a single collection campaign. However, machines are dynamic systems, and we likely didn’t capture their full complexity during an isolated measurement campaign. A predictive tool valid at a specific time may quickly become obsolete.

To address this, it’s crucial to implement a data collection system paired with performance monitoring to retrain the model regularly, if needed.

Conclusion

We’ve just explored, step by step, the deployment of a Machine Learning project in an industrial context. This project, though educational and intentionally simplified, highlights the many technical and strategic challenges of creating a predictive maintenance solution.

To go further, initiatives such as integrating continuous data or leveraging CMMS databases could not only enhance precision but also provide deeper insights into failure modes and prioritize interventions.

Feel free to share your feedback and experiences in the comments!

… Let’s rock with data! 🤘

In this new article, we will now translate these theoretical considerations into a practical case.

Description of the Dataset Used

In my projects, I consistently seek out datasets related to the pharmaceutical industry. In this case, as I couldn't find a suitable pharmaceutical dataset, I opted to illustrate this predictive maintenance project with a dataset from the aeronautics sector1. This choice is not problematic, as the principles of predictive maintenance easily transcend industry boundaries; methodologies developed for one field can often be successfully applied to others.

The dataset I selected tracks a fleet of about one hundred aircraft turbines, each equipped with 21 different sensors. A measurement is recorded for each of these 21 sensors over the operating cycles of the turbines. Thus, for each turbine, we obtain a multivariate time series.

In addition to sensor values, the training dataset includes a metric for each record called Time-To-Failure (TTF), which is the number of cycles remaining until the next failure occurs.

The Time-To-Failure is a numerical value, and there are two ways to handle it:

• We can retain its numeric nature, creating a regression problem where we aim to predict the number of cycles remaining until the next failure.

• Alternatively, we can transform this measure into a categorical variable with risk thresholds, creating a classification problem.

I chose the second option as it allows us to build simple, understandable operational rules. I thus transformed TTF into three categories:

• High failure risk (if TTF < 30 cycles),

• Moderate failure risk (if 30 ≤ TTF < 50 cycles),

• No failure risk, representing standard machine operation (if TTF ≥ 50 cycles).

The core objective of our project is to predict, based on sensor configurations, the risk zone in which the equipment is operating.

Analyzing the Dataset

What is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) forms one of the foundational steps of any data science project.

The tools we use to conduct EDA are often very simple: histograms, graphs, tables, etc2. However, this step, sometimes underestimated, is essential because it allows us to:

• Understand the structure and nature of our data,

• Visualize the distributions of different parameters,

• Check data quality, enabling us to clean missing or aberrant values as needed,

• Identify potential correlations between different sensors.

Indeed, the Data Scientist’s role when modeling a phenomenon is not limited to deploying a “standard” approach; it is always necessary to characterize the data and adapt preprocessing and modeling steps accordingly.

NB: The EDA process can be a bit tedious, and the purpose of this article is not to delve into it exhaustively. I will provide selected highlights here; the full EDA is available in this notebook on my GitHub repository3.

1. Data Types

A good initial practice when starting an analysis is always to display the variable types and basic statistics in data tables.

Conclusions:

• Our dataset is relatively clean and contains no missing values that need to be addressed.

• The sensors provide numerical data, with no categorical variables.

• Some parameters do not vary over the cycles, so I chose to exclude them from the study.

2. Univariate Statistical Analysis

Univariate analysis is a method used to examine the distribution of a variable in order to understand its fundamental characteristics, such as mean, median, or standard deviation. It allows us to detect trends, dispersions, and outliers, providing an overview of the parameter’s behavior in isolation.

Since the dataset contains only continuous variables, I displayed them as histograms. Here is the result:

Conclusions:

• The dataset does not contain significant outliers, except perhaps for sensor s6, which shows some atypical values.

• Some variables appear to follow a normal distribution, while others show strong asymmetry (see Kurtosis and Skewness scores4). Preprocessing, including correction (such as logarithmic transformation) and normalization, will be necessary before proceeding to the modeling phase.

3. Bivariate Statistical Analysis

Bivariate analysis examines the relationship between two variables to determine whether an association or correlation exists between them.

Below is the correlation matrix showing the existing correlations between each pair of parameters.

Conclusions:

• Several parameters show a strong correlation (positive in red or negative in blue) with our target variable, Time-To-Failure.

• Some parameters are strongly correlated with each other; for instance, s9 and s14 actually measure the same physical quantity.

We refer to this as multicollinearity when there are strong correlations between variables. This is an important point to consider for the next phases of the project, as several modeling algorithms are sensitive to this phenomenon, which can affect their performance and interpretability.

To address this phenomenon, it is often necessary to select the most representative features (variables) or apply dimensionality reduction techniques.

4. Principal Component Analysis

Given the complexity of a dataset with more than twenty columns, Principal Component Analysis (PCA) becomes a valuable tool for visualizing the underlying phenomena. NB: I previously covered this tool in the article on CPV 4.0.

PCA5 is a statistical method that, through projection into a new mathematical space, simplifies data complexity while retaining its essence. In other words, it reduces the number of variables to be studied, keeping those that best explain data variation.

As shown in the graph below, called a "Scree plot"6, PCA reveals that with only three principal components, we capture nearly 70% of the total data variance - with the first component alone explaining 53% of this variance.

By applying a 2D PCA, the results are as follows:

We observe a scatter plot, with each point corresponding to a different data record. When overlaying the failure risk on these visuals, a pattern emerges from the data:

• Records corresponding to optimal turbine performance (green) cluster distinctly on the left side of the chart.

• In contrast, records corresponding to pre-failure states (orange and red) gradually shift to the right.

Looking more closely, we even see distinct trajectories, revealing different degradation behaviors depending on the turbines.

Conclusion

The exploratory analysis phase highlighted and confirmed the potential of our data while validating the feasibility of our project: to detect early signs of failure. The dimensionality reduction achieved through PCA provided a clear view of equipment degradation dynamics.

In the final article in this series, we will explore how to model our process and create a concrete prediction tool capable of alerting maintenance teams before critical failures occur.

Feel free to check out the complete notebook on my GitHub to discover all EDA steps and share your feedback!

… Let’s rock with data … 🎸

Some Contextual Elements

Let’s start with the basics: what is predictive maintenance?

In industry, there are various types of maintenance1 :

• Corrective maintenance, which addresses and fixes a malfunction that occurs on a machine. This type of maintenance is considered reactive, meaning intervention happens after a resource failure.

• Preventive maintenance, which involves performing checks and/or replacing parts to anticipate and prevent breakdowns. This preventive approach can be executed in different ways:

  • Regularly and systematically over time, referred to as predictive or scheduled preventive maintenance.

  • Adaptively, based on data generated by the machine, known as conditional maintenance or preventive maintenance.

With these definitions, I think you now understand the importance of the issue.

Actions that are too close together may lead to unnecessary activities, while those too far apart may increase the risk of machine breakdowns.

The advantage of implementing predictive maintenance solutions is to trigger maintenance actions based on factual data only when the risk of a breakdown is significant. The challenge, therefore, is to determine the optimal alert thresholds to minimize overall maintenance costs2.

What Data to Collect?

The Key Role of Measurement Sensors

The core of predictive maintenance lies in collecting and analyzing data generated by equipment in operation.

Manual data entry is not suited to this purpose. The volume of information to be entered is considerable, and such entries would only introduce response delays, completely contrary to the tool’s philosophy.

Therefore, deploying such projects requires an initial technological investment. Machines must be equipped with IIoT (Industrial Internet of Things) sensors connected to a Cyber-Physical System (CPS) that centralizes the data in a database. Generally, SCADA systems3 or tools like Data Historian4 are used for collecting, storing, and aggregating machine data5.

What Types of Sensors to Use?

As shown in the diagram below6, there are many different types of sensors that can be used to characterize a machine's operation.

Common ones include:

• Vibration and acoustic (ultrasound) sensors,

• Power or electrical consumption sensors,

• Temperature and pressure sensors.

However, there’s no real limit: any operational parameter from equipment can feed a Machine Learning model (e.g., rotation, torque, flow rate). Discrete events, such as states, alarms, and logs, are also relevant variables to collect and analyze.

How Often to Analyze the Data?

Once sensor data is cleaned and stored, there are two main methods for analysis:

• Batch Processing: Aggregating data by cycle, day, or campaign, etc., and using a predictive Machine Learning model to make predictions based on these discontinuous data points.

• Real-Time Processing as a Time Series: Using anomaly detection algorithms, which often rely on neural networks and Deep Learning, to detect deviations from normal behavior.

There’s no right or wrong solution; it depends on the project objectives, the sensors available, machine downtime costs, etc.

Benefits of Predictive Maintenance

Predictive maintenance positively impacts several business aspects:

Here are the main KPIs that measure the gains from predictive maintenance7 :

• Equipment Availability and OEE (Overall Equipment Effectiveness)

  • Reduced unplanned downtime (fewer unexpected stops)

  • Reduced planned downtime (less total maintenance time)

• Maintenance Costs

  • Reduced repair costs (fewer breakdowns and severe failures)

  • Optimized resource utilization (better allocation of maintenance technicians and material resources)

• Quality Improvement

  • Well-maintained equipment is less likely to produce defects or quality variations. In the pharmaceutical industry, maintenance is regularly checked during audits and inspections.

Finally, here are a few quantified examples I found online8 :

• Predictive maintenance allowed SNCF Réseau’s teams to reduce switch incidents by 30%.

• AI-driven maintenance reduced costs by about 20% at Alstom.

• EDF monitors its wind turbines in real-time with IoT solutions, increasing equipment availability by 1% and reducing operating and maintenance costs by 2%9.

Project Charter

As always, to outline the project, I decided to formalize an AI Project Canvas10 to align as closely as possible with the industrial world. This tool specifies the project’s prerequisites, performance expectations, deployment considerations, and associated Return on Investment (ROI).

This project charter is illustrative and is based on approximations and calculation assumptions to be adjusted according to companies and their context.

An important point to consider in the ROI is the size of the fleet to cover. If several machines are equivalent, the development work will be more easily transferable from one machine to another.

Conclusion

I hope this introduction helped you understand the concept and purpose of predictive maintenance better.

In the next article, we’ll explore and analyze a maintenance dataset together.

… Waiting for that … Let’s rock with data … 🎸!

The prototype looks like this and can be accessed by following this link.

(PS: For transparency, I should mention that as the administrator of my Snowflake account, I have the ability to view the content of your queries… though I promise I won’t make a habit of it 😊)

The Web Application

I used the Streamlit platform to create the graphical interface for interacting with the AI. As I mentioned in the context of the gelose plate counting project, the interface isn’t particularly sleek, but it does allow us to quickly build functional applications.

I designed the interface to be organized in multiple tiles.

The Sidebar Navigation

The sidebar naturally displays the tool’s name and description, along with a range of options to customize how the chatbot operates.

The options allow users to configure settings such as:

• The language of the application: Responses from the model are typically provided in this selected language.

• The type of LLM used: I envisioned allowing users to choose between three different models: Llama3-8b, Mistral 7b, and gemma-7b.

• Model temperature: This parameter controls the degree of creativity or unpredictability in the responses. A lower value generates more deterministic responses, while a higher value encourages more creative responses.

• Memory of previous conversations: When using a chatbot, it’s beneficial for it to recall previous exchanges. This option includes a “Clear Cache and Reset” button in case users wish to clear the memory cache.

Suggested Questions

This is a common feature in user experience (UX) design for chatbots, as seen with major players like OpenAI1, Anthropic2, and Perplexity3. Providing question suggestions helps guide users through the tool and facilitates a smoother document consultation experience. I’ve personally set the first three introductory questions, and subsequent suggestions are generated by the LLM based on the last exchange.

Main Interface - The Chatbot

The main interface is incredibly simple and intuitive: it includes an input field and an assistant response field, much like ChatGPT.

A Critical Look at the Results

It’s time to evaluate the performance of the tool I created, and I’ll be completely honest 🧐.

Quality of the Responses

I had the opportunity to test the system informally with questions for which I already knew the answers. The level of hallucination is relatively low, with few errors or misunderstandings detected.

The primary flaw of the program is that the responses are often incomplete. It’s common for the LLM to respond:

• That it doesn’t know or can’t provide certain information, even though I know it actually possesses this information 😡

• By listing only one or two examples, when ideally a meta-analysis or a complete, exhaustive list would be preferable.

These points detract slightly from the user experience. I believe this is due to how I configured the RAG. By relying on only two or three information fragments to compose a response, the LLM misses the broader, high-level picture of the question. Revisiting the metadata linked to each fragment could enable it to execute queries like: “List all the devices…”

Does This Mean It’s a Failure?

I admit that the system falls short of my expectations, but “failure” is a relative term. The performance of an LLM-RAG system depends on three essential criteria:

1. Quality of Source Data

   I’m quite confident about the accuracy of the information available on the FDA’s site. However, I spent very little time preparing the documents in PDF format. It’s likely that images, tables, and specific layouts create obstacles to cleanly ingesting content into the vector database. For example: if you ever build an LLM-RAG, don’t expect to integrate PowerPoint slides—the result will be poor.

2. Investment

   Building a chatbot and processing queries is costly. I opted for relatively affordable embedding functions and models (note: I’ve spent around twenty dollars on this system’s implementation). If you want more effective, relevant tools, greater investment is necessary.

3. Time Spent

   I spent roughly 20 hours on this project, which is relatively little. I can’t afford to spend more time on educational and free projects. However, in AI projects, time is the ultimate resource needed to achieve high-performing models. It’s clear that results would improve by refining raw data, optimizing segmentation and vectorization, and enhancing indexing and retrieval methods within the vector database. However, all of these ideas require time and numerous iterations.

A more optimistic, less harsh view is to consider this project as a simple prototype, a first iteration… and it’s normal for a first iteration to not fully meet expectations.

The Value of the First Iteration

Many AI projects never see the light of day for various reasons:

• Poor understanding of user needs,

• Excessively long development times,

• Poor performance, etc.

One key tip for successfully executing an AI project is to release a first iteration as quickly as possible for testing. Even if it’s based on a very limited set of documents, and even if performance is lacking or the project’s scope is narrowly focused.

Early delivery to domain experts helps identify essential features, edge cases, and the most suitable performance metrics and tests. Exploring a prototype is always more effective than any brainstorming session for realigning a project.

Considering our project from this perspective, we can say that the 20 hours invested have met their objective 😁.

Improvement Paths

Finally, I’d like to share some improvement paths that could enhance the tool.

• Security Challenges

  One major challenge for conversational agents using RAG models lies in their resistance to prompt injection attacks4. These attacks exploit a model’s vulnerability by embedding hidden instructions within user queries. This can be dangerous, as malicious users could collect sensitive information and/or poison the system. Anticipating moderation aspects5 is crucial when developing AI.

  For my project, I applied minimal constraints to the LLM prompt; this would be insufficient in a professional setting.

• Optimizing the Document Base

  Only technical documents related to medical devices have been integrated into the document base. It would be beneficial to diversify the base by including documents with more of a “visionary” angle, such as theses, articles, normative texts on SaMD (Software as Medical Device)6, or even YouTube transcripts or conference proceedings.

• Capturing User Feedback

  To identify weak points in our chatbot, it’s essential to allow users to rate response relevance (typically with 👍 and 👎). This feedback can help align the document base with real user expectations and optimize prompts.

• Exploring Other RAG Variants

  The tool we’ve created corresponds to the most basic level of RAG (referred to as Naive RAG), but there are many variants to explore7. I’m particularly interested in trying out graphRAG8 — an idea for a future project.

Conclusion

In this article series, you’ve witnessed the birth of an LLM-RAG web application. Our newborn has taken its first cry, but there’s still much work to be done before it becomes a fully developed, reasonable, and high-performing adult! As with raising a child: testing, adjusting, iterating … these are the keys to a successful education!

I hope this journey through knowledge bases has inspired you with ideas to enrich your daily practice. Don’t hesitate to share your best use cases with me.

As always … Let’s Rock with Data … 🤘

How does a RAG work?

To illustrate the concept of RAG, I couldn't find a better visual than the one proposed by Aurimas Griciunas in his newsletter: Swirlai.com1. I will rely on it to explain the various stages of the project.

et's revisit the acronym RAG, which stands for Retrieval-Augmented Generation. In this acronym, we see two important facets:

• The “Generation” aspect, which involves the use of a Large Language Model (LLM).

• The “Retrieval” aspect, meaning information retrieval. It refers to building a library where we store fragments of information. When a question is asked, the first step is to retrieve the fragments of information relevant to the question and provide them to the LLM to help it construct a targeted response. This is why we speak of "augmentation," meaning enrichment, of the generative AI model’s answers.

Let’s go through this process step by step to better understand it.

Step 1: The initial creation of the vector database

The following steps are not performed routinely. They are completed once at the beginning of the project and during data update operations.

1.1. Document retrieval

I’ve already mentioned web scraping in a previous article. I used this technique again to search the FDA website for relevant documents. From the list of all AI-based medical devices, it is possible to retrieve all the “510(k) Number” case numbers and then bulk-download the associated PDF files.

1.2. Text fragmentation

Usually, documents are divided into several text segments, referred to as "chunks" (F). By doing this, we ensure that small fragments of text containing limited specific information are extracted. It's more effective to answer a precise question using small pieces of information.

As shown in the diagram below, we ensure that the fragments overlap to avoid edge effects, ensuring that a key sentence isn't arbitrarily cut in half.

1.3. Vectorizing text fragments

This part touches on the somewhat magical aspect of Data Science, but it’s possible to vectorize any word or text section. Vectorization transforms the text into a numerical representation that machines can process2.

This vector encapsulates the concepts, semantics, nuances, relationships, and more present in the text.

There are many different ways to vectorize text; for this project, I used the EMBED_TEXT_7683 function in Snowflake, which converts an English text into a 768-dimensional vector (C).

Thus, in the database, each document has both the text fragment (Chunk) and its vectorized equivalent (Chunk_vec). The result might seem obscure to humans but is highly structured and suited to machines.

1.4. Indexing in a vector database

Now that all our text fragments have been vectorized, they can be stored in a dedicated space called a vector database (D).

The vector itself determines its position in the database based on the numbers it comprises. The consequence (and this is exactly what we aim for) is that fragments of text with similar content will be stored close to each other, as illustrated by this figure4.

Step 2: The life of a query

With these fundamentals clear regarding the vector database, it’s relatively easy to understand what happens for each query. These operations are performed routinely whenever the LLM is called upon.

2.1. The initial query

As usual, we’ll pose the question to the LLM. This step remains unchanged.

2.2. Vectorizing the initial query

Just like the text fragments were vectorized to build our knowledge base, the query we address to the LLM will also be vectorized (E) using the same function as before (C). With mathematical techniques (like “Cosine similarity,” which measures similarity between two vectors) or clustering algorithms, we’ll search for the closest vectorized fragments to our query vector.

We can choose to select a variable number of fragments (2, 3, 5, etc.); this is particularly interesting to introduce nuance and diversity into the LLM's response.

2.3. Enriching the prompt

Once we’ve identified and selected the relevant chunks, we can enrich the initial query by feeding these chunks into the prompt. We then ask the LLM to specifically use the contextual elements we just provided to build its answer (B).

2.4. The LLM’s response

Nothing new for this step (A), which proceeds as usual.

Implementing it on Snowflake

For this project, I chose to use a platform called Snowflake5.

Snowflake is one of the titans in the Data world today, and I wanted to share a few insights about it (NB: this is factual, the article is not sponsored 😜!). It’s a platform that allows for storing, processing, and analyzing large amounts of data. Since it operates in the cloud, there’s no need to manage the hardware or software infrastructure.

Snowflake is known for its ability to handle all stages of the data lifecycle on a single platform6. Here’s how it acts as an orchestrator:

• Data ingestion: Snowflake enables data collection, whether structured (like databases) or unstructured (like files, images, or data from third-party applications).

• Data storage: Once data is collected, it can be stored in either a data warehouse or a data lake, depending on its nature and format.

• Data manipulation: Snowflake offers tools to transform, analyze data, or even apply AI models. A notable feature is that computing tasks are independent of data storage, allowing different teams to work simultaneously without disrupting other processes.

• Data redistribution: After manipulation, the data can be made available as results for third-party applications, end users, or transferred to other systems.

In addition to these features, Snowflake has layers to manage metadata, access rights, logs, budgets, etc. Snowflake also promotes its scalability, meaning its computing power adjusts to demand, as do operating costs.

On the downside, one major issue is the cost, which can quickly rise, especially if data isn’t rationalized or project strategy isn’t well-defined.

Personally, I enjoy using Snowflake because it’s relatively simple and allows me to create clean, industrial-grade solutions in Data Engineering (which involves all the steps for making data available). Data Engineering can be overwhelming with many tools and specialized programming languages: Spark, Scala, Rust, dbt, etc. 🤯

Having a single, scalable solution that works with Python and SQL allows me to build prototypes within a few hours, which suits me perfectly.

For this project, I heavily relied on the following tutorial from Snowflake: Build a Retrieval Augmented Generation (RAG) based LLM assistant using Streamlit and Snowflake Cortex7.

Conclusion

Our journey into the dark and jargon-filled side of the force has come to an end (no more “embedding” and “vector databases”🤯) ! I hope I didn’t lose you along the way.

Even if it's a bit demanding, I was keen to break down the various components of an LLM-RAG. I truly believe these tools will become pervasive in the professional world, and it's crucial to understand how they work and the scientific foundations behind them.

Indeed, when we break the apparent magic of these LLMs, we’re better able to grasp their limitations and implicitly place them in their proper context: as practical tools, not oracles of truth.

In the final part of this series, I will present the demo version of my application, and you’ll be able to test it yourself.

Let’s rock with Data 🤘… and “RAG” (-a-muffin) 😜

Introduction

We often see the visual below to illustrate the interconnection between different AI technologies1.

This diagram shows that Generative AI is an integral part of Deep Learning and Machine Learning. Within the circle corresponding to generative AI, we obviously find LLMs, which rely on very specific neural network structures called transformers. Their unique mechanisms give them specific properties.

Generally, I simplify things by explaining that:

• The role of ML and DL models is to transform data into information; the goal is usually to predict phenomena or support decision-making with a factual basis.

• On the other hand, Gen AI models, particularly for text processing, can enrich the information to make it "information ++".

It’s a bit of a metaphor! What I mean is that by asking an AI to summarize a text, extract an action plan, or answer specific questions, we concentrate the information—the value—that was previously diluted in the text.

However, LLMs present certain challenges, especially regarding reliability. One of the main issues, inherent to how these models work, is what we trivially call hallucinations2. Since these models are trained on a vast dataset from the internet, they often lack precision when asked specific questions.

Try this test: ask a standard LLM to:

• Create a training plan for your team.

• List the challenges faced during the development of effervescent paracetamol.

• Produce a risk analysis for cleaning validations on your site.

In most cases, the result is quite disappointing, providing content that seems plausible but is devoid of substance and specificity.

To improve relevance, these systems need to be customized by injecting external, specific knowledge from your business context.

Data Injection Techniques

There are many ways to customize an LLM, but we will focus on two main families:

1. Fine-tuning: This involves taking a pre-trained model and retraining it on new personalized data. This allows the LLM to better understand your sector-specific concepts. However, this approach also risks distorting the model, slightly altering its overall skills in favor of its specialization, as some of the model’s underlying parameters are modified and replaced.

2. Retrieval-Augmented Generation (RAG): This method combines the native capabilities of a pre-trained LLM with a real-time information retrieval system. The information is vectorized and stored in an external database, and the LLM fetches relevant data to format it. The primary advantage of RAG is that it doesn’t modify the model's base structure, so there’s no risk of altering its syntactic capabilities3.

To conclude with a metaphor: imagine our LLM is a chef specializing in French cuisine:

• Fine-tuning would be like sending the chef for a one-year training in Chinese cuisine. They would learn new recipes but might lose some of their basic techniques and gestures.

• RAG would be like providing the chef with a giant library of Chinese cookbooks, allowing them to explore Chinese subtleties while retaining their French culinary knowledge.

In our case, I’ve chosen the RAG method because it is relatively simple and quick to implement on a private server. The risk of degrading the model is absent, avoiding additional complexity.

The LLM-RAG Duo: What are the Pharmaceutical Applications?

Here are a few use cases scattered throughout the drug lifecycle:

1. Technology Watch in R&D and Regulatory Monitoring

With these tools, you can quickly and automatically scan a corpus of texts such as scientific articles, patents, or official documents. This allows you to manipulate, connect, and summarize information, even if it’s spread across different documents. LLM-RAG systems are increasingly used to identify new indications for known active substances, known as drug repurposing.

To illustrate this use case for regulatory aspects, one could cite the Ring platform4 available on the A3P website5.

2. Knowledge Management

Knowledge management6 is a key process in the pharmaceutical industry, involving capturing, organizing, storing, and sharing accumulated knowledge throughout a product’s lifecycle (from R&D to commercialization). In reality, this is a complex task requiring a highly mature organization, as it is often difficult to break down silos between different organizations, which do not work with the same documents or follow the same timelines.

An LLM-RAG duo appears as a perfectly suited solution to bridge these different data sources7.

3. Organizing Computer Servers

In my early industrial years, I had the opportunity to deploy the 5S methodology8, which involves physically organizing production workshops and work areas. Spoiler alert! Often our digital files are worse than our workshops, and it’s very difficult to find key information. LLM-RAG systems can slice through and explore massive amounts of documents, with the luxury of bypassing language barriers, as they can integrate any original language.

4. Post-Marketing Surveillance and Pharmacovigilance

LLM-RAG systems can continuously monitor adverse event reports, analyze external databases such as PubMed, and assess emerging risks by integrating up-to-date information. This allows for rapid responses to safety alerts.

In short, I’ve limited myself to four examples, but I could have listed many more: recurrence analysis, information extraction, knowledge preservation, training, etc.

Once again, the only limit is our imagination, as there are numerous topics where we seek to cross-reference and refine information to convert it into structured knowledge (= "Wisdom" in the diagram below) :

Building a Regulatory Knowledge Base

I will now describe the project we will undertake in this series of articles. I’ve decided to create a knowledge base, which I will associate with a conversational agent (or chatbot), on the subject of regulatory submissions for medical devices.

I chose this topic because the data is easily accessible as open data on the FDA website9. Over 950 medical devices using artificial intelligence are listed on the site. These can include diagnostic tools, radiography analysis devices, cardiovascular medical devices, etc.

For each of them, a PDF summary is available, which constitutes an overview of the submission file validated by FDA teams (description, therapeutic indications, training data, verification tests, etc.). The content is thus verified and of high quality.

Conclusion

I hope this article has clarified the key concepts of LLMs and RAG.

The next article will be a bit more technical, as I will show you how to retrieve, slice, vectorize, and inject nuggets of knowledge into a RAG architecture. The exercise will be conducted using the Snowflake platform.

… Let’s Rock with Data … and RAG-a-muffin 🤘

In statistical jargon, a "multivariate" analysis (sometimes referred to as MVDA for MultiVariate Data Analysis) refers to the study of several variables simultaneously in order to understand the relationships between them. Multivariate analysis allows us to highlight interactions, correlations, and patterns that would not be observable by examining the variables individually.

How to conduct a multivariate analysis?

Here’s how I structured the page dedicated to multivariate analysis in CPV 4.0.

As you can see, the left sidebar has evolved. Previously, it was possible to select certain quality attributes to monitor them specifically. This individual monitoring is no longer relevant, since we are now moving to a holistic monitoring of the entire process from maintenance onward.

The available panel allows for the selection of:

• a time window

• a product/dosage

• an analysis scope

I also envisioned the possibility of performing the analysis on different scopes by selecting: CMA, CPP, and/or CQA.

Let’s Start Simply: Correlation Analysis

The upper part of our interface serves as an introduction to the MVDA. It contains two charts:

• A heatmap:

The heatmap is a two-dimensional chart where the correlation coefficient between two variables is displayed. This is why we see a diagonal made up of values of “1”.

• If the coefficient is equal to 0, it means that the two factors are independent of each other.

• If the coefficient is positive, it means that an increase in variable x1 induces an increase in variable x2.

• If the coefficient is negative, it means that an increase in variable x1 induces a decrease in variable x2.

Of course, the closer the coefficient is to -1 or 1, the stronger the intensity of the correlation. In addition to this diagram, there is a hierarchical clustering tree, or dendrogram1. This tree groups variables into blocks that are close to each other. With this tool, we have a preliminary view of the dynamics at play in our dataset.

• A scatter plot:

As we just saw, the heatmap summarizes the intensity of the relationship between two factors in a numerical value, but it doesn’t provide more information about the shape of this correlation. For this reason, it is possible to select, on the right-hand side, the two variables of interest to visualize their distribution as a scatter plot.

The two charts I have just described do not actually constitute an MVDA. However, I still chose to implement them, because the need to visualize simple correlations is a healthy and recommended first step toward more advanced analyses.

A First MVDA Tool - Clustering

Clustering is a data analysis technique that groups a set of points into homogeneous sub-groups called clusters2.

Here is a graphical representation of a clustering; each point corresponds to a production batch:

This chart allows us to visualize what could be called "the degree of overall similarity" that exists between the batches. Batches that are close to each other in this chart resemble each other across all their characteristics.

A Quick Aside: To obtain this type of chart, a data transformation step must first be performed; this is why the x and y axes have such peculiar names. This transformation method is a statistical method I am particularly fond of, called PCA (Principal Component Analysis). I won't go into detail about this methodology, as it deserves an entire article on its own. I will just explain its basic concepts.

The complete dataset we are working with contains 23 variables3. The principle of PCA is to apply dimensionality reduction to the dataset, meaning that we artificially create new axes, called principal components, and project our entire dataset onto this new two-dimensional space4.

By projecting 23 columns onto a two-dimensional space, we intuitively understand that some of the subtlety hidden in our data will be lost. It’s a bit like compressing a photograph and accepting a slight blur. To use another analogy, I often describe PCA as a roadmap, where only the major roads are presented. It’s a simplified way of perceiving reality while ignoring certain superficial details.

To learn more, I invite you to watch the following french video5.

A Commented Example: By selecting data related to the product named “Product 1,” we see that there are four to five different clusters within the population of batches.

The analysis becomes even more relevant when this visualization is enriched by displaying additional variables. For example, one could overlay value gradients on a clustering diagram, or even cross a clustering diagram with references to raw materials.

In this illustrated example, one can, for instance, observe a clear link between the API reference and the impurity level. From experience, this type of analysis is far more useful for guiding investigations than a statistical test or a boxplot diagram, which can often be misleading as they do not reveal the “shape” of the data.

A Second MVDA Tool - Hotelling's T²

The first tool we just discussed is well suited for visualizing and investigating disparities, but it is not the ideal visualization for alerting on deviations and managing a process.

For this reason, I have implemented another visualization, which is based on a statistical test: Hotelling’s T²67.

Hotelling’s T² is the multivariate equivalent of the Student’s t-test but applied simultaneously to all the variables. This score allows us to compare, using a single criterion, all the parameters of a batch to the average vector of the parameters of other batches. This makes it possible to detect if a batch behaves differently from all the others, considering all variables at once.

An Illustrated Example: Here is a simplified yet illustrative example proving that, with the T² technique, it is possible to visualize variations that cannot be detected on our traditional control charts.

The red point below clearly appears as an outlier. However, there is a high probability that it would not be identified in individual histograms, as it is not an extreme value in the various distributions.

With Hotelling’s T², we can clearly see that this aberrant point differs from the others. And it becomes even clearer when this point is plotted on a control chart with a threshold corresponding to the 95% confidence interval.

It happens that certain CQAs show atypical values, and it is never easy to determine whether these deviations are isolated or not. With the T² method, it becomes much more comfortable to assess whether a batch is truly atypical.

The Contribution of Machine Learning

This article has already covered many advanced concepts. Therefore, I will not provide a detailed review of all the charts on the “Machine Learning & Prediction” page. Instead, I will just explain why machine learning is a relevant tool for CPV.

Machine learning involves creating predictive models, based on real data, that link the different variables together. Regardless of the type of algorithm used, it is possible, after training, to display a diagram showing the importance of the various variables (or “Feature Importance”).

One of the major benefits of machine learning is the ability to prioritize the contribution of different variables in predicting a CQA.

I find this particularly interesting because it gives more depth to our investigations and action plans. This allows us to prioritize improvement efforts in an objective, science-based manner. I believe this is a significant advantage in risk management and aligns perfectly with the approach described in the ICH guidelines8.

The evaluation of the risk to quality should be based on scientific knowledge and ultimately link to the protection of the patient.

The level of effort, formality and documentation of the quality risk management process should be commensurate with the level of risk.

Conclusion

We have reached the end of a project that I loved working on, as it stands at the crossroads of everything I enjoy about my profession. I hope that, through this journey into graphs and numbers, I have convinced you of the value of data-driven process control.

There were two facets to this project related to quality, each bringing its own set of improvements:

1. Digitalization in the service of CPV process efficiency and performance.

2. Digitalization in the service of knowledge and understanding of the targeted process.

I hope I succeeded in combining all these criteria within the same tool!

Does this philosophy resonate with your approach to data analysis and process optimization? Feel free to share your experiences and thoughts in the comments.

Let’s Rock with Data to improve CPVs 🎶 🤘

In statistical jargon, "univariate" analysis refers to focusing on the variations and distribution of a single variable. Typically, this is the foundation for most CPV analyses, which is already very informative. We will see, however, how to optimize it to make its use more relevant and efficient.

The Two Graphical Pillars of CPV

The CPV approach is mainly based on two types of visualizations1:

• The Control Chart:

A control chart is a statistical tool used in quality management to monitor and control a production process. It helps detect abnormal variations in the process by graphically comparing the chronological evolution of a quality characteristic against predefined control limits. Depending on the requirements, the time axis can be represented in terms of dates or batch chronology. The main objective of this tool is to visualize drifts.

The displayed limits correspond to the specifications (in red), i.e., the required standards, and the lower (LCL for Lower Control Limit) and upper (UCL for Upper Control Limit) statistical limits (in blue), defined by ±3 standard deviations from the mean.

• The Capability Chart:

A capability chart is a visual tool used in quality management to assess whether a production process is "capable" of producing results that meet the required specifications. It compares the natural variation of the variable with the tolerances, thereby determining whether the process provides consistent and reliable results. A capable process is one that is both tight and centered around the specifications.

The capability of a process is measured by the Cp and Cpk indices. In general, a process is considered capable when these indices are greater than 1.3.

These two types of charts, which are very classic, are logically at the heart of the CPV 4.0 tool. However, I have envisioned special features to enhance them.

The Options

• Visualization Options

I propose two different visualization options: quarterly visualization (since it is often the timeframe used for periodic process analysis), as well as the opportunity to display or not display the LCL and UCL.

These two options were chosen to illustrate frequent use cases; however, many other options could be considered. When using open-source tools like R and Python, it is possible to achieve total customization of the charts and visuals2. This way, one exceeds the capabilities of all commercial software, and any need can be custom programmed. Thus, the end user defines their needs and how they wish to manipulate and display their information.

• Outlier Detection

Typically, control charts are accompanied by outlier visualization using the Nelson Rules3. Nelson's rules are a set of criteria used to detect non-random variations in control charts. They include eight different rules, each characterizing specific patterns of points on the control chart.

In addition to these classic Nelson rules, I chose to add an alternative that I find more elegant, which is based on a machine learning model: an anomaly detection algorithm called Isolation Forest45. This method detects anomalies by creating random partitions in the data until each point is isolated (and, by definition, outliers are easier to isolate than others, which is how they are identified).

I find this visualization complementary in terms of alerts because it really focuses on sudden and drastic changes in the CQAs. Nelson's rules primarily allow a retrospective analysis of observed behaviors. In contrast, a predictive algorithm learns from past data and can identify unusual variations from the first excursion. If we aim for responsiveness, this type of tool is appropriate6.

A Custom Interpretation

I wanted the tool to offer users a literal analysis of the two charts described above.

Rather than using generative AI—which would have been very entertaining 😜 but difficult to control—I preferred to build a flowchart that generates predefined text based on certain conditions. The text thus adapts to the reality of the data.

It is then up to the user to complete and balance this very descriptive analysis with their own explanatory elements.

The Ultimate Feature?

To ensure maximum time savings, I have provided the possibility to export the charts, as well as the conclusion text, as a ".docx" file, which will directly feed CPV, APQR, or investigation reports. I designed a very simple ".docx" report for this demonstration version, but once again, it is fully customizable. One can design modular custom reports with any chart, table, text block, etc.

Conclusion

The "Univariate Analysis" interface of CPV 4.0 meets the objectives I set for myself: all the markers of a classic CPV analysis are there, and this information can be obtained in just a few clicks.

Once the tool is customized according to the specific needs of the teams, it becomes possible to generate a report in just a few tens of minutes after carrying out some routine checks and adding a few pieces of information.

Teams can now focus their energy on what really matters: explaining the variations, identifying root causes, and implementing the right actions. This corresponds to steps 2 and 3, and that’s what we will share in the final part of the journey 😎.

Does this philosophy resonate with your approach to data analysis and process optimization? Feel free to share your experiences and thoughts in the comments.

Let’s Rock with Data and CPVs 🎶

Introduction

To begin with, here is a brief introduction for readers who may be less familiar with the pharmaceutical field.

The pharmaceutical sector is characterized by its highly regulated nature. Regulatory texts and health authorities require manufacturers to guarantee and prove that manufacturing processes are controlled, meaning they deliver the expected results.

When deploying a new production process, an initial validation approach is undertaken to ensure it works correctly. However, an initial validation is not sufficient; it is also necessary to ensure that the system does not drift and remains in a state of constant control throughout the product's lifecycle12.

This is precisely the purpose of CPVs (Continued Process Verification).

CPVs involve the systematic collection and analysis of data on production components and processes to ensure that product outcomes remain within predefined quality limits.3

In practice, this means that certain critical variables for product quality will be monitored and compared to their specifications. The analysis is generally communicated to the various departments of industrial sites to share the degree of process control and define potential actions to be taken.

Typically, a CPV report is prepared periodically for each product or product family. Thus, on large sites, it is not uncommon to have several dozen documents to prepare annually. The volume of administrative work is therefore very significant.

I conclude this context setting with some terminology related to CPVs that will appear in the series of articles:

• A CQA (Critical Quality Attribute) is a physical, chemical, biological, or microbiological property of a product that must be controlled to ensure the quality and safety of the drug.

• A CPP (Critical Process Parameter) is a manufacturing process parameter whose variability can significantly impact a CQA and must therefore be monitored or controlled to ensure product quality.

• A CMA (Critical Material Attribute) is a physical, chemical, biological, or microbiological characteristic of a raw material that can influence a CQA and must be monitored or controlled to ensure product quality.

• An OOS/OOT (Out-Of-Specification/Trend) refers to a test result that does not meet the predefined specifications for a product, raw material, or process.

Traditional CPVs VS “CPV 4.0”

In the digital age, new opportunities arise to modernize this quality process. A number of companies are transitioning from the classic CPV approach to a more modern, data-driven approach.

I have represented these two symbolic worlds that coexist on a diagram, with the different dimensions that distinguish them.

This type of visual is simplistic, but it still allows for effectively assessing an organization's maturity in managing CPVs.

Typical Pain Points of the Traditional Approach

1. A generally tedious preparation process

Currently, many companies prepare their CPV analyses using generalist statistical tools, such as Excel or industrial statistical software (e.g., Minitab®, Tibco Statistica®, …). Although these tools allow for optimizations through macros, they often require repetitive tasks and many manual manipulations to collect, input, and format the data. These manual tasks also increase the risk of human error.

In practice, it is not uncommon to spend several dozen hours preparing a single report. Moreover, a major problem with the traditional approach is its lack of scalability: the time required to create documents increases proportionally with the number of documents to be produced.

CPV 4.0 Solution : With a well-structured data model and good data governance, I estimate that the preparation time for a report can be reduced by about 80%. Furthermore, this time savings through automation makes the process scalable, as once the software is developed, the total time required to prepare 10 or 20 reports is almost the same.

A CPV 4.0 process allows teams to focus on value-added activities (problem-solving, improvement actions, …) instead of office tasks.

2. A sometimes rigid and too retrospective process

Given the dozens of hours needed to prepare a report, CPV writing is often done in a staggered manner, following a fixed schedule. This leads to a retrospective analysis; a long delay may occur between the appearance of production drifts and when they are analyzed. This approach prolongs investigations, delays corrective and preventive actions, and causes a loss of opportunity to optimize the process.

CPV 4.0 Solution : Data processing is done almost in real time. It becomes possible to set up automatic alerts to react quickly. With Machine Learning (ML) tools, weak signals can even be captured, and risky situations can be predicted before they materialize into OOS4. Indeed, machine learning algorithms are capable of detecting subtle patterns and emerging trends well before they are noticeable using traditional methods.

A CPV 4.0 process allows for quick reactions to contain drifts and their impacts. Reactivity = Reliability.

3. A process focused on a handful of CQAs

Due to a lack of resources, CPV programs often focus on a limited set of CPPs and CQAs. This approach, while effective for detecting major drifts, has a significant limitation: it increases the risk of missing subtle interactions between different variables. By focusing only on a few terminal indicators, we risk overlooking the inherent complexity of pharmaceutical manufacturing processes.

Moreover, in the coming decades, the amount of exploitable data will only increase. This is the trend of history, and it would be regrettable not to consider these information reservoirs on the processes.

CPV 4.0 Solution : Using multivariate analysis techniques, it becomes possible to take into account a wide range of variables simultaneously. This provides a more complete and nuanced view of the complex interactions within the production cycle. Better understanding of materials and processes helps reduce the risk of non-compliance and batch rejections.

A CPV 4.0 process allows for the development of a clear and efficient product strategy.

4. Tools that are not very interactive

Control charts and capability graphs5, which are at the heart of CPV analysis, are good tools for summary reports. However, these graphs have several important limitations:

• Their ability to visualize processes remains very high-level, making it difficult to really understand one batch among its neighbors.

• These tools are static: they do not allow for data manipulation or cross-referencing, which slows down and limits the depth of exchanges and investigations. Often, team interactivity around data is a driver of collective intelligence.

CPV 4.0 Solution : From the beginning of this article, I have mainly been talking about technical and statistical aspects, but in reality, the CPV philosophy is not limited to tools and numbers: it is primarily based on cross-functional collaboration between teams. Modern solutions offer better transparency and interaction with data. This reduces dependency on subjective judgments, improves knowledge sharing, and achieves better collective alignment on actions.

A CPV 4.0 process makes data alive and interactive.

With the framework well set, it's time to move on to the project.

The Project Charter

For this third project, I propose an application that illustrates all the points I have developed above. I have called this application CPV 4.06 ; it is the implementation in code of the vision I have of a modern CPV process, applied to a dry forms manufacturing process.

To do this, I used a dataset shared by the Žagar team et al. in one of their publications7. You will find in this document the detailed description of the dataset.

As usual, to contextualize the project, I propose a project charter summarizing its scope, as well as how to calculate an ROI (Return on Investment) for this type of initiative.

I remind you that the figures communicated in this document are purely illustrative.

Some Clarifications on the Project Scope

• This project requires technical prerequisites in terms of infrastructure and data governance.

  To implement the type of tool that I will present, it is necessary to first structure different data sources within a Data Warehouse architecture. The latter being connected to a LIMS, an ERP, and production data management systems (IPC, SCADA, …). I will not discuss these points further, as they are upstream of our project.

• The tool I develop involves computer system validation, which I will not describe in the upcoming articles.

  One small clarification: my CPV 4.0 tool is enhanced with some AI features, but these tools are simply there to highlight insights and catalyze the decision-making process.

  However, analyses and decisions remain human. Therefore, there is no issue with AI validation as such.

• Finally, I do not necessarily recommend a systematic CPV 4.0 approach in every situation.

  Indeed, there are scenarios where traditional CPVs work very well and are fully sufficient. KISS: Keep It Simple Simple!

  I see interest in this approach particularly for:

  • high value-added products

  • highly multi-factor processes (multi-step processes, biopharmaceuticals, …)

  • processes identified as low-capable, where troubleshooting needs are significant

Conclusion

Through this article, I hope I have demonstrated that CPVs provide a tremendous opportunity for digitalization. The journey is long, and there are many steps to bridge the gap between the traditional CPV approach and the modern 4.0 approach.

The next article will focus on the first of these steps. I will show you how I managed to automate statistical analysis and report creation.

I strongly encourage you to explore the CPV 4.0 app now to get an idea 😃. Your feedback will be valuable to help us further improve it. So, don't hesitate, test it, and share your impressions!

Let’s Rock with Data to improve CPVs 🎶 🤘

This final part of our series will be more concise and visual. I will take you behind the scenes of the dashboard creation process that will close this project. I will also share with you some of the tips I frequently use to build dashboards with Microsoft’s tool.

Here is the link to the "Workplace accidents analyzer" dashboard so you can review it alongside the reading.

Interface Description

My initial objectives were as follows. I wanted:

• A synthetic tracking tool that fits on a single page, allowing for quick and intuitive consultation by teams.

• A very visual and graphical communication, consistent with the subject of personnel safety (avoiding long texts and data tables).

• Simplified reading focused on 4 or 5 key graphs.

I chose to organize the dashboard using a fairly classic structure with three main blocks, laid out from left to right:

• A sidebar containing the various filtering options. This allows the analysis to be divided based on the available main criteria (industry, sector, categories, etc.).

• A left section focused more on the victims: age, gender, and injury mapping. It primarily includes an accident count, an age pyramid, and a human body diagram.

• A right section dedicated to describing the accidents: their nature, the tools involved, and the root causes, when specified.

Tips #1: Properly Structure Your Data Model

The data was clean after running my Python script since I had spent time and energy mastering the data output format and its structure. Thanks to careful data preparation in advance, the setup phase in Power BI was extremely quick.

However, I always recommend structuring your data well in Power BI by creating logically interconnected tables. A classic and effective way to handle data is by using the following architecture:

• A central table, often referred to as the "fact table." In our case, this is the table containing specific data for each accident.

• "Dimension tables," which contain information, categories, etc., that may repeat across accidents.

As you can see, this is referred to as a star schema.

With this organizational structure, there are fewer errors when creating visuals, and working with large datasets becomes much faster.

For more explanations, I refer you to the following link1.

Tips #2: Look for Strong and Impactful Graphs

A dashboard must be clear and functional. It must adapt to the habits of the teams.

If a particular type of graph is essential for your field of activity, it's worth trying to reproduce it. In the case of workplace accidents, it's very common to include a human body visual in the various entry forms to quickly point out affected areas. So, I took on the challenge of reproducing this type of diagram.

I found the site synoptic.design2, which makes it easy to create custom visuals, and I used this tutorial3 to create mine. It’s very easy, for example, to create engaging visuals that add a playful and intuitive dimension to dashboards.

Tips #3: Pay Attention to Your Visuals

We often put a lot of effort into creating AI tools or dashboards. It is therefore crucial that users embrace them, adopt them, and want to interact with them. In this regard, design and aesthetics play a significant role.

For the visual aspect, my personal routine is often as follows:

• I think about the overall structure of the dashboard. The visuals will differ for a concise summary or for a four or five-page interface.

• I browse royalty-free image banks for inspiration regarding the colors and/or structure of the dashboard. For this project, I came across this image that I found interesting and from which I retained the color code.

  

• I quickly create a background image in PowerPoint by pre-positioning blocks (often in light tones with rounded corners). This method allows for proper alignment of sections and avoids overloading the Power BI file with numerous objects (like images, text boxes, etc.).

  

• I write a few hex color codes on a post-it note to reuse them in future charts. PS: It’s worth checking that the colors are inclusive and appropriate for people with color blindness.

• I quickly build a logo and a title. I also favor icons, in the spirit of an infographic, to avoid large descriptive text blocks.

Aesthetic preferences can differ from one person to another, and it's possible the visual won’t appeal to everyone. However, with these tips, you're not just creating a simple dashboard but offering a professional "data product," functional and visually appealing, that will meet your users' needs.

Tips #4: Optimize the Relationships/Filters Between Your Visuals

Most Power BI visuals are interactive, meaning it’s possible to click on the charts to perform actions.

By going to "Modeling" → "Manage Relationships," you can decide how a visual interacts with its neighbors. You can use charts to filter or subdivide neighboring charts. This is done very quickly and is often very useful in a practical use case.

Some of My Other Projects

I’d like to take the opportunity to reshare a few of my previous dashboards:

• Explora’thesis: A dashboard about students' doctoral theses in pharmacy.

  

• FDA unofficial dashboard: A synthetic dashboard based on open data from the FDA.

  

• Pharma sales monitoring: A dashboard tracking the sales figures of a marketing team (using simulated data).

  

Conclusion

Through this project, I wanted to show managers and industry experts that it is now possible to retrieve and manipulate unformatted information. This information is often hidden, diluted in the sea of all our daily tools and favorite databases 😉.

You can now bring it to life!

I undertook this project on workplace accidents because I found the subject motivating, but I remind you that all this is transferable and can be applied to any type of business.

The only limits are your business needs and your imagination!

After reading this, I invite you to think about your own sources of unstructured textual data. Do you have any other use case ideas for the pharmaceutical industry? Feel free to share them in the comments!

Stay tuned for the next project, and… Let’s Rock with LLM/Data 🤘.

(As a reminder, LLM stands for: Large Language Model)

I want to warn the reader that this second part will be slightly more technical as I will reveal some programming elements.

First Step: Data Collection from the INRS Website

The EPICEA database1 is available online but cannot be downloaded as-is. The first part of the work was to retrieve (also known as scraping in technical language) the data in a usable format. Web scraping is a technique that allows you to automatically collect information from websites using a program or script, somewhat like copying and pasting information on a large scale but in an automated way.

This technique involves using a program or script to navigate through web pages and extract specific pieces of information from the HTML code, such as text, images, files, or other more or less structured data.

In our case, the web scraping strategy was carried out in two distinct steps:

• First, searching for the accident file numbers available in the EPICEA database.

• Then, extracting case-by-case specific information for each accident.

Disclaimer: It is important when collecting data from the internet to check the legal aspects to stay compliant with GDPR regulations (General Data Protection Regulation) applicable to personal data2.

In this case, there is no issue since:

• INRS allows the use of its database for non-commercial purposes.

• The database is completely anonymized.

• We will only work on a sample of this database, focusing on the pharmaceutical, agri-food, and chemical industries.

I wanted to mention this scraping phase, but I won’t elaborate further in this article as it’s not the core of the process. I invite you to check out Chapter 2 of the notebook3, if you are interested in the implementation mechanics.

Second Step: Deploying an AI Language Model Securely

▶ Installation

Throughout this project, I really wanted to demonstrate that it is possible to reconcile the use of generative AI with data confidentiality.

To use a language model locally while avoiding data leaks, I chose to use Ollama4. This is an open-source platform designed to run language models locally. It allows users to download, run, and interact with various pre-trained language models without requiring an expensive cloud infrastructure.

The installation process is described in their technical documentation5.

Note: For those using Windows, it is generally recommended to install the Windows for Linux WSL system6. A new version executable on Windows has been released, but I have not tested its stability.

▶The first query

Once these two operations are completed, it’s possible to open the WSL command interface and type the following command:

ollama run NODEL_NAME

Replace MODEL_NAME with the name of the model you wish to use. The table below lists all the models available at this time7.

Once the model is downloaded, you can query it directly via the command prompt or call it directly in our Python programs using the Ollama client with the following piece of code:

from ollama import Client
from langchain.callbacks import get_openai_callback
from langchain_openai import ChatOpenAI
from langchain_mistralai import ChatMistralAI, MistralAIEmbeddings
from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate
from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.messages import HumanMessage

nom_modele = "mistral"

# Configuration du modèle Ollama
llm = ChatOllama(
    model=nom_modele, 
    format="json",
    temperature=0,
    top_k=10,
    top_p=0.9,
    repeat_penalty=1.1
)

Third Step: Querying the LLM in a Controlled Manner

As described in the previous article, there are many precautions to take with LLMs to limit their inherent risk of hallucinations.

Without going through the entire code, I will highlight three security measures that help guide and control the AI model's behavior.

1. A Python class named “Accident”, which lists each of the items we want to extract:

class Accident(BaseModel):

    Metier: str = Field(description="Metier, role ou fonction de la victime ayant subi l accident.")

    Sexe: str =  Field(description="Sexe (Homme, Femme) de la victime ayant subi l accident.")

    Age: int = Field(description="Age de la victime ayant subi l accident.")
     
    Type_accident: str = Field(description="Type d'accident survenu. 1 ou 2 mots maximum.")

    Blessure: str = Field(description="Descriptif médical des blessures ou symptomes. 1 ou 2 mots maximum.")
        
    Deces: bool = Field(description="La victime est mentionnee comme decedee ou morte.")

    Circulation: bool = Field(description="Accident lie a la circulation.")

    Malaise: bool = Field(description="Accident lie a un malaise type AVC, infarctus, accident cardiaque.")

    Suicide: bool = Field(description="Accident lie a un suicide.")
    
    Machine: List[str] = Field(description="Machines, pièces ou objets impliques dans l accident. 1 ou 2 mots maximum.")
                                  
    Cause: List[str] =  Field(description="Facteurs ayant cause directement ou ayant favorisé l'accident. 1 a 3 mots maximum par facteur.")

    Zone: List[str] =  Field(description="Zone du corps humain impactée par l'accident")

You can see that for each item to be extracted, I provided a descriptive sentence (sorry it is in french 🙂) and a format for the expected result. The format can be a string (str), an integer (int), a boolean (bool), a list (List), etc.

The output format is particularly important because it helps guide and constrain the LLM in retrieving the specific information.

2. A prompt, which is a structured instruction similar to what you would write to interact with ChatGPT online:

template_string = """Tu es un analyste qui relit des compte-rendu d'accidents et effectue des saisies. 
Analyse le texte ci-dessous qui se trouve entre les triples apostrophes et extrais-en les informations requises. 

Descriptif de l'accident : ```{descriptif}```

Si l'information n'apparait nulle part dans le narratif, n'invente rien.
Tes réponses doivent être en français exclusivement et doivent être les plus concises et précises possibles. 

IMPORTANT: Ta réponse DOIT être un objet JSON valide, respectant strictement le schéma suivant. N'inclus AUCUN texte en dehors de cet objet JSON.

{format_instructions}
"""

For reference, the section named '''{description}''' in the prompt will automatically be replaced by the text corresponding to a specific accident.

Moreover, you can adjust a model parameter called temperature, which roughly translates to the model's level of creativity in responding to the original prompt. By setting it to 0, we maximize the likelihood that the LLM will strictly follow the prompt and avoid taking too many liberties.

3. Post-hoc verification functions

Despite all the precautions provided by the first two security measures, we cannot completely rule out receiving erratic responses. For this reason, it's recommended to create verification functions that will ensure the response matches the expected format and, if necessary, correct the value in question.

I will share an example I encountered: for some unknown reason, one of the LLMs provided the result “young boy” when I specifically asked it to choose between two possible answers, “man” or “woman.” These deviations are rare but can happen, so they should be anticipated.

I invite you to check out Chapter 3 of the notebook for more details.

Summary of This Third Step

The diagram below illustrates the concept of the three security measures.

I am very satisfied with the extraction work performed by the tool. In fact, after applying these precautionary measures, format errors are extremely rare. Beyond the form, we can now focus on the content and accuracy of the responses.

Last step : Select the better LLM model

As we saw earlier, several types of models exist. They each have their own architecture, number of parameters, and training text corpus. There are no inherently good or bad models, and we cannot predict their performance a priori.

The best approach is to remain pragmatic and compare them, challenge them with practical cases, and select the ones that seem best suited to our problem.

For this project, I chose three candidate models: Llama3.18, mistral9, zephyr10 , whose characteristics are described in the table below.

All three are open-source models (goodbye ChatGPT 4o 🫡).

Since I chose to run these models on my personal PC, cost won’t be a significant factor. In an industrial context, however, it would be a crucial consideration.

The benchmarking protocol is simple: to assess the performance of these models, I personally analyzed about thirty accident reports. I will simply compare the number of correct answers provided by the three candidate models with reality.

The result of this benchmark favors the Mistral model.

Conclusion

Below is a randomly selected example showing the result obtained with the Mistral model. The result is more than satisfactory.

Through this article, I wanted to shed light on how an LLM works and how to interact with it in an automated and controlled way.

This was a slightly more demanding article on a technical level. I hope it didn’t lose you too much 😉. Don’t worry, the next article will be simpler and shorter, presenting the Power BI dashboard I created to finalize this project.

Feel free to share your thoughts in the comments or share your experiences with using LLMs!

Stay tuned, and … Let’s Rock with Data 🎶

According to the website of the Ministry of Health and Solidarity:

“Every day, two people die at work, and more than a hundred are seriously injured.”2

When you've worked for several years in an industrial environment, you understand how crucial the topic of workplace safety is. This inspired me to develop a project around the study of workplace accidents.

During my research, I found the EPICEA3 database, which belongs to INRS, an institute responsible for preventing risks in the professional environment. The EPICEA database lists over 21,000 cases of serious workplace accidents since the 1990s. In this database, there are "narratives," meaning accident reports written in a completely freeform manner.

A technical challenge arises: how can we extract and manipulate information buried in freely written text?

Before answering this question, I’d like to share a quick reminder.

Structured VS Unstructured Data

When people think of data, most envision tables of numbers (like Excel) or structured databases in rows and columns. This is what we call structured data.

On the other hand, there is unstructured data, which doesn’t follow a predefined format or schema. Their size and content vary. Some examples of unstructured information:

• Images and videos,

• Text documents (articles, reports, publications, emails, etc.),

• Audio files,

• Genetic sequences,

• Spectrograms, etc.

The following infographic summarizes the fundamental differences between these two worlds4 :

It shows that 80% of company data is in a freeform format5.

Obviously, healthcare companies are no exception to this observation. Many IT tools rely, at least partially, on free-text fields. Here are some everyday examples of unformatted data in the pharmaceutical industry:

… And this list is far from exhaustive. Every company holds deposits of untapped data because manual analysis would be too tedious.

How can we refine these mines of raw data?

For this project, we will see how to transform unstructured information from freeform text into data tables, allowing us to visualize and analyze them. Technically, this task is known as named entity extraction.

To carry out this activity, we will use LLMs (Large Language Models)6. These are AI models trained on vast amounts of text and designed to understand and generate human language. The most well-known example to the public is ChatGPT, but there are many others.

The advantages are numerous:

• The tools are adaptable. Data entry habits vary from person to person, or from site to site, depending on culture, organization, and history. By focusing on context, an LLM compensates for these variations and standardizes the text.

• It’s possible to overcome the language barrier, as most tools are capable of integrating multiple languages (with better results for more common languages).

• By using a generalist model already pre-trained, we avoid the tedious step of text annotation and training. The final performance might be slightly lower, but we save a significant amount of time.

What are the risks of LLMs in the workplace?

There are many risks associated with using LLMs in companies. I've listed a few of these risks and the measures I plan to take to manage them in this project.

Confidentiality

• Users may expose sensitive and confidential information to LLMs, creating a risk of data leakage.

For this project, I downloaded several models and ran them locally on my computer, rather than online through an API. Thus, none of the information I submit to these models is transmitted to unknown servers; in fact, the code can even be executed without any internet connection.

This ensures a private and secure language model that does not compromise data confidentiality.

The only downside to working locally is that you don’t necessarily have significant computing power, and the inference times (calculation time required to obtain a prediction) are relatively long. This is especially true if the analyzed text is massive or if the model has a large number of parameters.

Reliability

• LLMs have a strong tendency to hallucinate and frequently generate inaccurate or fabricated information.

The principle of a generative AI for text is to predict the next word based on the context and the history of words in its possession. These tools, therefore, lack real understanding and logic. The risk of hallucination is structural and intrinsic; it can be reduced but cannot be entirely eliminated7.

These tools are Swiss army knives that can be used in various ways, and the risk of hallucination varies depending on the use case. Asking an LLM to translate or summarize a text is different from asking it to create a new text from scratch.

In our project, I will ask the model to identify specific information within a text. The risk of hallucination is logically lower because the scope for creativity is limited.

Finally, there is a whole field of expertise called prompt engineering, which focuses on optimizing requests to improve output quality. It is also possible to adjust certain technical parameters, such as the temperature, which controls the model’s level of creativity.

Security

• LLMs present potential vulnerabilities to external attacks.

LLMs deployed in companies are often self-learning structures. Since the goal is to interact with them, they must remember the history of a conversation and adapt their behavior to be more relevant, etc.

This advantage is also a danger because users, through repeated solicitations, can modify the chatbot's settings. This is how we’ve seen conversational agents be withdrawn from the market after being literally corrupted by their users' requests8.

In our project, the user will not interact directly with the LLM. The prompts are written by me in the source code, and the user will not be able to alter them.

Conclusion

LLMs allow us to uncover hidden treasures within unstructured data. Thanks to them, analysis opportunities that were once inaccessible due to time or resource constraints become possible. However, these tools are double-edged: as promising as the benefits are, the risks must be taken seriously!

The next article will be more technical, with code to explain my technological choices and share some concrete results.

What do you think? Do you have other ideas for use cases regarding unstructured data? Share your experience in the comments!

Stay tuned, and … Let’s Rock with Data 🤘

It is time to share with you “AI SMART Gelose Counter”, the web application that I have built. Here is the link to test it in parallel of your reading.

Deploying AI in Production: A Commonly Underestimated Challenge

It is relatively "easy" to explore a dataset on your own or to code parts of an application. However, deploying an AI in a production environment is far more complex than it seems. Let’s explore together why so many projects fail and how to avoid these pitfalls.

Around 80% of AI projects fail to be deployed in the field, which is twice the failure rate of traditional IT projects.1

Among these 80%, a significant number of AI projects fail because they do not meet the practical realities of the field. In some cases, AI performance is lacking, while in others, they don't fully meet user needs or fail to integrate well into existing workflows.

Building a data product, mastering its deployment phase, and ensuring its adoption by users requires a lot of communication and collaboration. It’s more a matter of project management and organization than technology. A good practice is to provide AI prototypes early on so they can be quickly tested and challenged.

In this prototyping game, Streamlit2 is a particularly effective Python framework!

What is Streamlit?

Streamlit is a Python library that makes it easy to create interactive web applications for data science and machine learning.

Its design is minimalist, but the advantages are numerous:

• Collaboration and sharing: All team members have access to the same version of the model. They can thus visualize and interact directly with the data and/or the model. By bringing the application to life, users are involved from the very early stages of development. This helps gather feedback and integrate it directly into future iterations to better meet the real needs of the teams.

• Operational efficiency: It takes only a few hours to create a data product because the process is greatly simplified for developers. Streamlit significantly streamlines the process for developers, sparing them from complex tasks like managing user interfaces or deploying applications.

• User simplicity: A simple interface hides the code in the background. When coding a program and wanting to share it in raw form with a colleague, there are often numerous packages to install. This can be a real headache and not something you can ask field operators to handle. With a Streamlit web application, you create a vehicle, a container, that allows the code to be executed immediately with no hassle.

In short, working on deployment with a prototyping logic promotes:

• Right-First-Time

• Time-To-Market

• User adoption

Let’s take a look at how Streamlit fits into the project architecture.

A Few Words on the Architecture of My Solution

The top part of the diagram corresponds to the training phase, as described in the second article. As a reminder, training takes time, so it is a one-off activity, and there’s no question of retraining the program each time it’s used. What we want to do routinely is use the trained model to infer a new image, meaning to make a counting prediction on a new petri dish.

Streamlit, despite its many advantages, has one limitation: it is not possible to install large models that require a lot of storage space and memory. Streamlit only handles what we call the "front-end", the storefront of our application.

For the back-end, meaning the model execution and calculations, we use an API. An API is a tool that allows different applications to communicate easily with each other by exchanging information securely. In my case, I used FastAPI3, which I deployed on a Heroku server.

The API acts like an automaton: if you submit a .jpg or .png image to the address https://ufc-counter-api-e72d4934bdd3.herokuapp.com/predict, it will execute the model and return a list of coordinates containing the identified bacterial colonies.

Description of the Streamlit Application

AI SMART Gelose Counter is relatively simple and goes straight to the point.

The steps are as follows:

1. Uploading a petri dish photo, either by dragging and dropping it into the “Drag and Drop” area if you have one handy 😉 or by selecting one of the demo photos from the “Sample Library”.

2. The predicted image then appears in the center of the screen. Two options enhance the visualization experience:

   • The “Activate shutter view” option allows the user to hide the “green boxes” surrounding the colonies.

   • The “Show probabilities” option displays the probabilities associated with the detection of each colony. The closer the probability is to 1, the more confident the model is that it has identified a "colony-forming unit".

     These two options are particularly useful for model verification and validation activities.

3. The total colony count appears in the right sidebar. The total number of colonies is compared to the actual value, at least for the sample library photos that I counted myself.

This temporary interface, with no frills, will do exactly what is expected of it: allow subject matter experts to challenge the model and the features of the software.

Going Further: Improvement Ideas

I coded and wrote this project in just a few dozen hours; there’s always room for improvement. I’d like to conclude by sharing a few improvement ideas that come to mind:

• I limited myself to creating a counting model, which considers all colonies as a single object. With more data, it would have been possible to build an identification algorithm that could predict the type of bacteria based on the external appearance of the colony.

• I tested only one model, Yolov5; it would have been interesting to benchmark several models to find one with better detection capability.

• Finally, as I mentioned earlier, the dataset was objectively unbalanced. It could have been useful to carry out a “Data Augmentation” step. This consists of creating new images for the most underrepresented classes by making slight variations (rotations, zooms, contrasts, etc.). Increasing the volume of available images generally improves training.

Conclusion

I hope you enjoyed this first experience with Computer Vision applied to the world of quality control laboratories. As for me, I’ve rediscovered the very fun world of microbiology from a new angle! 😛

Feel free to test the application and share your feedback! Whether you're an expert or a novice, your opinion matters to me.

Stay tuned, and … Let’s Rock with Data 🤟

#AIinMicrobiology #ComputerVision #DataScience #DeepLearning #Gelose #Counting

In this article, we will explore, in a simplified way, the structure of a deep learning model and how to train a neural network to perform a specific task.

What is machine learning?

There are many real-world cases where it is impossible to explicitly program a machine. Many tasks, especially in computer vision, cannot be reduced to a list of rules. In our case, related to image analysis, it is impossible to create mathematical equations that define in every way what a bacterial colony is.

Instead of coding explicit operating rules, the principle of machine learning is to "train" a program to perform a task using large amounts of data. The system is then trained to recognize patterns in the provided information in order to make predictions or decisions on new data.

Humans no longer provide logical instructions but instead supply the machine with labeled data, i.e., raw data associated with the outcome to predict.

For our project, we will provide the program with:

• Pictures of agar (a solid culture medium used for bacterial growth) on which bacterial colonies are present.

• A table containing the positions of these colonies.

The application of deep learning to images

When dealing with images, it is common to use specific models known as neural networks and we are talking about deep learning. I highly recommend the excellent video by 3blue1brown1 that explains very well how these structures work and how they perceive images.

For a neural network, an image is simply a matrix of numbers, where each cell corresponds to a specific pixel.

In short, neural networks are composed of various layers of artificial neurons. Each neuron is interconnected with its neighbors, and each connection is assigned a weight that can be adjusted.

By feeding the neural network numerous images whose nature is known, and by slightly adjusting the weight of each connection whenever an error is made, the various parameters will converge towards an optimum.

Through this process, called training, the neural network will "itself" identify areas of interest and the important characteristics of the images. Once trained, the model can generalize its knowledge to make predictions or decisions on new data.

This is, of course, a simplified view behind how a neural network works. In reality, things are more complex, and many choices must be made regarding the network architecture, the mathematical functions involved, and the training parameters.

For this project, I chose to use YOLOv52, a deep learning model specialized in real-time object detection, using a convolutional neural network (CNN) architecture.

YOLOv5 stands out for its speed and efficiency, while offering high performance with a lighter and more optimized architecture compared to its predecessors. It is not the most powerful model, but it strikes a good balance between performance and execution/memory efficiency.

The most critical item: the dataset

As we mentioned earlier, training is based on input data; it is this information that will shape the model's parameters and its ability to generalize.

The dataset is the cornerstone of any machine learning or deep learning project!

The quality and relevance of the AI we are going to create directly depend on the richness, diversity, and representativeness of the training data. Without a carefully selected and prepared dataset, even the most sophisticated neural network architecture will only produce mediocre or biased results.

The dataset I used for this project is the one used in the article by Makrai et al3. It is composed of 369 images on which domain experts have marked the coordinates of 56,865 colonies.

Therefore, I took a detailed look at the dataset composition.

This dataset is not perfect; some strains are overrepresented compared to others, and blood agar culture media are predominant. As a result, there is a risk that our model will perform better at counting colonies on blood agar than on other culture media. In an industrial context, it would be good to populate new images and label additional data.

For our part, we will continue training with the existing images.

Results after training

Our deep learning model is ready! 👨🏼‍🍳

When tested on new images, the magic happens, and we obtain the following results.

The model's accuracy is 92%, which is encouraging, even though it remains below the objectives set in the project charter.

Don’t miss the next article, where you’ll be able to test our bacterial detection application yourself!

Stay tuned, and … Let’s Rock with Data 🎶 and Bacteria 🦠

The complete code is available on my GitHub: https://github.com/arnaud-dg/UFC-counter

Introduction and Context

One of the core activities in a microbiologist's profession is interpreting analyses from samples taken during production; these can be raw materials, finished products, or environmental samples to control the cleanliness of premises.

These analyses are primarily presented in the form of agar plate cultures; sometimes colloquially referred to as "Petri dishes."

In France, there are thousands of microbiology laboratories, including those in pharmaceutical and food industries, hospitals, universities, etc. The website supermicrobiologistes.fr lists at least a thousand of them on its interactive map [1]. With each laboratory potentially performing dozens of analyses daily, the number of agar plates read nationwide is absolutely massive!

There are two types of results when reading an agar plate:

• Quantification of the number of bacterial colonies (=CFU), referred to as counting or enumeration. A CFU (Colony Forming Unit) is a measure used in microbiology to estimate the number of viable microorganisms in a sample.

• The nature of the germs that have grown on the culture medium, referred to as identification.

An Innovative Project: AI in Service of Bacterial Colony counting

This first educational project aims to study a solution to automate the "counting" part using Artificial Intelligence. It was inspired by an article in Nature Scientific Data written by Makrai et al. [2]

What are the advantages of automatic counting systems:

1. Enumeration is sometimes a tedious step as it involves visually counting a number of points on a given surface. While in most cases contamination is moderate, there are much more complex scenarios where it is more significant. An automated solution would theoretically save precious time on reading heavily contaminated agar plates.

2. There is also a risk of bias or human error during counting, especially for highly contaminated plates. Consequently, a reproducibility increase could also be a supplementary benefit.

3. Finally, I believe that these tools allowing accelerated reading also offer new opportunities in working methods and organization. We could imagine building double checks, early readings, monitoring of bacterial growth rate over time, etc. on this basis. This opens up a range of possibilities.

Digital counting tools, allowing colonies to be pointed out on a screen, already exist. Some methodologies had already been explored in an A3P article [3]. However, these tools that facilitate counting with a grid and/or rely on image contrast do not solve all problems.

In this case, it seemed relevant to me to use Artificial Intelligence, because detecting whether a point is a bacterial colony is more complex than it appears. Indeed:

• There can be many false positives: air bubbles, condensation droplets, human writings, external stains, etc.

• The appearance of colonies is highly variable (round, fluffy, etc.), and multiple colonies can coexist.

• There may be low contrast between the color of the colonies and the color of the support.

• Some colonies will be difficult to count because they are located at the edge of the plate or overlapping each other.

• ... etc.

For all these reasons, I decided to use Computer Vision tools [4], which allow training a model to recognize the external appearance of a colony.

The Project Charter

When building digital tools, it's important to plan the project well and focus on the notion of value. The idea is to use Artificial Intelligence only if it is necessary and if there is a return on investment (=ROI) at stake.

To properly identify the necessary resources, well describe the use of the tool, and ensure that it brings real added value, I have grouped the different information in a document. This type of document is called a "Lean Canvas" or an "AI Project Canvas" [5] and it provides a 360° overview of a digital project.

Some Important disclaimers

I conclude this article with a few points that seem important to specify before continuing.

• This project is a case study, therefore the information in the project charter, especially financial information, is purely fictional; the document simply serves to illustrate the project.

• Through this project, I absolutely do not promote the fact to replace the microbiologists with automated reading. Reading an agar plate requires expertise. In my philosophy, the purpose of AI tools is not to replace experts, but to reduce workplace difficulties and improve the organization of our laboratories. To quote Garry Kasparov [6]:

  AI Should Augment Human Intelligence, Not Replace It

• This project is what we call a PoC (Proof Of Concept). The goal is to create a prototype together; it will absolutely not be a finished product, which would involve hundreds of hours of teamwork. Be lenient on the result.

The next article will popularize the methodology used to train a Deep Learning model from a dataset of agar plate images.

Stay tuned, and ... Let's Rock with Data 🎶 and bacteria 🦠

All questions and comments are fully welcome in the comments section.

Sources

[1] https://supermicrobiologistes.fr/la-carte-des-labos-de-microbiologie/

[2] Makrai, L., Fodróczy, B., Nagy, S.Á. et al. Annotated dataset for deep-learning-based bacterial colony detection. Sci Data 10, 497 (2023). https://doi.org/10.1038/s41597-023-02404-8 https://www.nature.com/articles/s41597-023-02404-8

[3] https://www.a3p.org/scanstation-analyse-microbiologique/

[4] https://en.wikipedia.org/wiki/Computer_vision

[5] https://towardsdatascience.com/introducing-the-ai-project-canvas-e88e29eb7024

[6] https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it