Data science for the public good

1. Aims and objectives

At the Office for National Statistics (ONS), automatic classification of survey responses to Standard Industrial Classifications (SIC) and Standard Occupational Classifications (SOC) currently uses the matching tool GCODE, developed by Statistics Canada. For Census 2021, the ONS developed an in-house tool. The in-house tool uses rules-based, probabilistic methodology that is similar to GCODE, and relies on matching to comprehensive knowledge bases.

The ONS tool was used to successfully classify Census data to SIC and SOC and is used by the Labour Force Survey (LFS) and the Coronavirus (COVID-19) Infection Survey (CIS). Match rates of approximately 65% for SOC and 55% for SIC are being achieved by the tool, at an accuracy rate of over 90% as reported by the product owners.

Whilst accuracy rates are high, the match rate is much lower, with a significant number of records that cannot be automatically classified requiring manual processing by expert coders.

We investigated whether a knowledge-based approach or a data-centric machine learning (ML) one would be more performant as a global solution. This means that consideration needs to be given to the variety of use-cases it could be built for. Accuracy at all hierarchical levels of coding would be the desired output with the solution’s operational cost efficiency to be taken into consideration. This means that the cost of developing, running and maintaining the tool as well as the cost of manual coding needs to be taken into consideration in evaluating solutions.

Therefore, our objectives for this project are to prototype a solution that:

1.       Is accurate:

• yields confident and accurate SIC and SOC classifications
• yields accurately classified data that can be stored and relied upon in the future

2.       Is portable:

• can run on any platform without requiring specialist hardware

3.       Is cost-efficient to:

• build
• test
• maintain

4.      Minimises the need for manual coding

5.      Provides a way to refer the uncodable or unknown records to manual (human coders)

2. Our approach

We propose a solution using machine learning, which aims to capture the power and potential of quality data. For the purposes of simplicity and ease of explanation, we opted for a basic approach using Logistic Regression (LR) applied to a frequency matrix based on uni-grams, bi-grams and tri-grams commonly known as a Term Frequency – Inverse Document Frequency (TF-IDF) representation. More details can be found in the methodology sections. However, we would be keen to try more advanced AI approaches in the future as mentioned in the conclusions.

3. Data collection

The following datasets are used as benchmarks for our experiments between the two proposed solutions.

Census

The Census dataset consists of half a million Standard Occupational Classifications (SOC). We placed greater emphasis on SOC results in this report.

Labour Force Survey (LFS)

A survey of employment circumstances, yielding 300,000 samples of SOC codes gold-coded to the SOC 2020 standard.

Coronavirus (COVID-19) Infection Survey (CIS)

70,000 SOC gold-coded samples from the first four to five weeks of the survey.

Annual Survey of Hours and Earnings (ASHE)

A small set of 819 SOC gold-coded samples.

Destination of Leavers from Higher Education (DLHE) survey

A small set of 879 SOC gold-coded samples.

4. Results and discussion

We compare the machine learning (ML) and knowledge base coding tool methods on match rate, overall accuracy and matched accuracy when applied to these gold-coded datasets. By match rate we refer to the percentage of samples that each approach can confidently classify. By confidently we mean that the probability for a record being of a certain class exceeds an agreed confidence threshold. Matched accuracy refers to the accuracy within the matched subset. All benchmarks that we display in this section refer to validation sets we kept in reserve during ML training.

Match Rate

The % of entries that have been assigned a viable SOC/SIC code e.g. match rate = ((12-2)/12)*100 = 83.3%

Matched Accuracy

The % of entries assigned the correct code out of those assigned a viable SOC/SIC code e.g. match accuracy = (4/(12-2))*100 = 40.0%

Overall Accuracy

The % of entries assigned a correct code out of all records (in ML’s case, before thresholding) e.g. overall accuracy = (4/12)*100 = 33%

The census and Labour Force Survey (LFS) models

Of the 500,000 gold-coded census records, we split 360,000 for training, 64,000 for testing and 75,000 for validation. The selected text features were “job_title”, “occ_job_desc” (the respondent describing their work activities), “business” (business area of the respondent), and “employer text” (name of employing company). The match rate on the validation set was achieved for a target 95% accuracy rate using a global threshold of 0.6. In Table 1 we see the results for both Standard Industrial Classifications (SIC) and Standard Occupational Classifications (SOC) codes for census. The machine learning (ML) tool performs best for both SIC and SOC codes in our comparison tests (see Table 1).

The LFS data consisted of 300,000 records with SOC codes gold-coded to the SOC 2020 standard. The data was split 217,000 training data, 38,000 test and 45,000 validation. For our predictions we used the columns “OCCT” (occupation title), “OCCD” (occupation description), “INDT” (industry title) and “INDD” (industry description). The match rate was again achieved with a 95% target using a global threshold of 0.6.

The results on the test sets (SOC) were 84% overall accuracy for census and 85% overall accuracy for LFS, which incidentally are the same as the validation set results within rounding error. Similarly SIC codes test and validation results were the same at 81%. This indicates that the training sets available had enough samples to learn a universal pattern and that the resulting models could not be biased to the training datasets.

Table 1 Overall benchmark results between ML and coding tool

Table 2 displays results for the Census SOC codes per class for the 10 most populated classes and the last row in the table is the overall result for all classes for both tools. Again, we see the machine learning tool outperforming the existing coding tool in every testing aspect.

The third and fourth columns (ML, Coding tool) show the percentage of the class that each tool could correctly classify and on the fifth and sixth columns (ML Only, Coding tool only) we can see the percentage of correct classifications made by each tool only (for example, ML only 67.8%, means that 67.8% of samples on the validation set could only be correctly classified from the ML tool and not the coding one). There we see that for the coding tool the exclusive correct classifications were in the range of 0.2% to 3.1%, whereas for the ML tool the corresponding range was 20.5% to 87.9%. We made this benchmark to understand whether a hybrid solution would be suitable.

Table 2 Benchmarks per class for the 10 most populated classes. The last row ( overall) is calculated using all available classes

Recall by Class – Census Occupation

The Coding Tool Only range is 0.2 to 2.6 this time and ML only ranges from 14% to 70.5%.

In Table 3 we see similar patterns that we observed on the LFS dataset, where the ML solution clearly outperforms. The Coding Tool Only range is 0.2 to 2.6 this time and ML only ranges from 14% to 70.5% .

Table 3 LFS benchmark results per class for the 10 largest classes. The last row (overall) is calculated using all available classes.

Recall by Class – Labour Force Survey (LFS)

The hybrid models

The hybrid models were trained with a mixture of LFS and census data. These models performed .

The hybrid models were trained with 0.5M census training data and 0.3M LFS data split into 0.68M train and 0.12M test data, an 85:15 split. Once trained, each model was used to classify the validation sets for CIS, ASHE, DLHE, which consisted of 64,000, 819 and 879 gold-coded samples respectively. The following table demonstrates the training data and validation data text fields for each of the models created:

Table 4 Training and validation text features for the hybrid models for CIS, ASHE, DLHE

The benchmark results are appended in the following table. The resulting human in the loop split was achieved using a variable threshold with a target accuracy of 0.975 on the train-test dataset.

Table 5 Benchmark results for DLHE, ASHE, and CIS for the coding tool and the hybrid Logistic Regression models. The numbers in bold are the highest scores for each category. The blue highlighted numbers are scores without using a match rate threshold, ie. 100% match rate.

In Table 5 we see that for DLHE the Logistic Regression model outperformed the coding tool in all aspects. For ASHE, we see that the matched accuracy is similar, however, the match rate for the LR model is 11% higher and the overall accuracy 25% higher for LR.

Finally, for the Coronavirus (COVID-19) Infection Survey, LR outperforms the coding tool on match rate by 19% and the coding tool demonstrated a 98% matched accuracy compared with 90% for LR. However, again the overall accuracy was higher for the LR model by 30%, and the CIS implementation of the coding tool is unusual, having disabled nearly all fuzzy-matching functionality. Disabling this functionality meant that the tool would only classify direct matches, which in turn results in much higher accuracy than usual, at the cost of match rate, which dropped at 47% compared with 66% for the LR model. The performance for both solutions was lowest on DLHE.

Table 6 A – Recall by Class – Covid Infection Survey (CIS)

Table 6 B – Recall by Class – Annual Survey of Hours and Earnings (ASHE)

Table 6 C – Recall by Class – Destination of Leavers from Higher Education (DLHE)

Table 6 A,B and C shows benchmark results for the 10 largest classes for CIS, ASHE and DLHE. The last row (overall) is calculated using all available classes. The two highlighted rows are the only ones where the coding tool outperformed the ML one.

In Table 6 A,B and C, LR outperforms the coding tool in all of the most populated classes that we benchmarked, apart from one – class 4122 (book-keepers and payroll managers) where the coding tool clearly outperforms LR by a large factor, 17.9% for CIS and 11.8% for ASHE.

Figure 3 gives a graphic overview of the two tools’ performance across different classes, for different datasets. The orange curve represents the number of samples in a specific class in log scale. The classes are sorted by size from the smallest to the largest. The cyan (light blue) curve represents the number of records within a class that the LR matches correctly and the blue one  (dark blue)  the same for the coding tool.

We observe that the cyan curve for the LR model stays relatively close to the orange curve, however, this is not the case for the blue curve (coding tool), which tends to stay far apart with occasional big drops. The larger the drop, the larger the inability for the tool to predict the associated class.

Figure 3: Per class comparison overview for all classes and for all datasets. The vertical axis is the number of correct classifications for the two tools in comparison. The all curve represents the number of rows per class. The horizontal axis represents the different classes, from the least populated (left) to the most (right)

Confidence plots

Figure 4 shows a graphic representation of the confidence of predictions for the LR model in log scale. The horizontal axis is the prediction probability and the vertical represents density, that is, how many right or wrong predictions were made with the associated probability range/bin. The blue bars represent correct classifications and the red bars represent erroneous ones.

The observation we can make on these graphs and especially for the three larger datasets is that there is a clear pattern where the higher the probability the higher the likelihood for a correct prediction, and the other way around. This means that we could make decisions in regard to probability thresholds that could lead to higher accuracy expectations. The pattern is not so strong for our smaller datasets, however, we could argue that this could be because of the size of the datasets being fewer than 900 samples, but we see a similar trend for probabilities over 0.8 and lower than 0.3.

Figure 4 Confidence density plots for the LR tool in Log-scale

Human in the loop

The option to mix human expert classification and automation is desirable for this project, hence we demonstrate how that can be achieved with a machine learning (ML) system. Initially we opt for a global probability threshold, before expanding to class-by-class solutions.

Global threshold

Figure 5 demonstrates the relationship between match rate, matched accuracy, and general accuracy for each of our datasets for the Logistic Regression (LR) tool.

The green curve overall accuracy is the number of correct classifications divided by all samples, whereas matched accuracy is the number of correct classifications divided by the number of matched samples. In this figure we see a similar pattern, with better results for census, Labour Force Survey (LFS) and Coronavirus (COVID-19) Infection Survey (CIS) and a bit worse for the Annual Survey of Hours and Earnings (ASHE) Looking at this graph, it seems that we could pick a desired probability threshold that could steer our results on future unknown sets towards the target accuracy values. We used these plots to choose a probability threshold of 0.6 for census and LFS models.

Figure 5 A graphical representation of match rate, accuracy and matched accuracy for the LR model and all datasets.

Variable threshold

As an expansion to this idea, we implemented a method of generating probability thresholds by class that aim for a particular accuracy target. Potentially, the model could be more confident in its classifications of some classes than of others and this could be reflected in a per-class threshold map. The aim of this variable threshold is to provide match rates and matched accuracies such that the matched accuracy would correspond to a higher match rate than the one we could get from a global threshold.

Our experiments show that this technique works well for our transfer learning examples (CIS, ASHE, DLHE models) but not so well for census and LFS, where the results were very similar to – and rarely better than – the ones obtained with global thresholds. Another issue we had with this technique was that, although we would calibrate it aiming for 95% matched accuracy, our results never quite matched it, but we got 90% for CIS, 86% for ASHE and 81% for DLHE as we saw in Table 5. The fact that it is a transfer learning exercise could well explain the loss of accuracy.

5. Methods

The machine learning (ML) classification tool consists of a nested pipeline made of three major pipelines: text processing, feature engineering and model generation. The code for the general tool was partially adapted from ACE2, a general-purpose text classification pipeline developed by the Data Science Campus specifically for this class of problems.

Text processing pipeline.

Text processing consists of the following tasks:

• punctuation removal
• lowercase
• spell check
• split joined up words
• stopword removal
• stemming

Regarding stemming, we chose this over lemmatization as it seems more robust and faster because it is calculated rather than being looked up in lemmatization dictionaries.

Feature engineering

This step consists of embedding the text into a feature matrix prior to model training. We use a TF-IDF (Term Frequency – Inverse Document Frequency) representation of text. We find that having a dedicated TF-IDF matrix per text-field gives us better results than simply merging all text fields and creating a single matrix. Separate matrices allow for different semantics and frequencies between text fields. The TF-IDF matrices were generated for unigrams, bigrams and trigrams for lowest frequency of five documents, then concatenated into a single feature matrix.

Model generation (Logistic Regression with TF-IDF)

Logistic Regression (LR) was chosen for its simplicity, explain-ability, and efficiency when it comes to text classification. The Logistic Regression model, is as simple as learning a weights vector θ_i and bias b, to satisfy the equation:

$$ {\hat{y}}_i = (\theta_i) \ X + b  $$

Where y_i is the prediction probability for each class i, and X is the features (tf-idf) matrix. The weights vector is learned through established and known first or second order optimization methods, in our case LBFGS (Limited Memory for the Broyden–Fletcher–Goldfarb–Shanno algorithm). The objective function for the optimisation was chosen to be the log-likelihood, which is the standard for the Logistic Regression algorithm.

6. Calibrating the match rate

After the model is trained, it is evaluated on an unseen test set in order to assess the model’s ability to generalise. This is when we also estimate the per-class probability thresholds. This is achieved by creating a tuple of probabilities and prediction outcomes (that is, correct, false). Then we pick the maximum sequence whose correct classifications account for a ratio equal or higher to our target accuracy. The resulting threshold map is saved together with the model.

7. Conclusions

As an alternative tool for Standard Industrial Classifications (SIC) and Standard Occupational Classifications (SOC), a machine learning (ML) approach performs extremely well and is naturally suited to this classification problem. Delegation of low-confidence records to manual coders is easily achieved, and the significant accuracy gains ML makes over expert systems using lookup lists should translate to monetary savings against the requirement for manual coders.

A concern for statisticians in some outputs is extremely high accuracy in confirmed matches, given the reliance of multiple statistical products upon the results of the coding system. Figure 5 suggests that in these situations, for sources with suitably large training sets (census and Labour Force Survey (LFS)), near-100% accuracy with a match rate of approximately 50% can be achieved with a high global probability threshold of 0.9. This would still reduce required manual coding, with great confidence in the machine-coded portion of the output. Performance after transfer to other datasets is generally good, except in cases where the survey population is significantly deviant from the training data (that is, Destination of Leavers from Higher Education (DLHE); in which case both approaches performed poorly).

A self-evident caveat to this is that gold-coded training data must exist. While gold-coded SOC records exist, the same is not true of SIC and this would have to be addressed. While ML will tolerate random errors in its training data (for example, occasional mistakes in chosen codes) where systematic errors exist, say, for example, persistently miscoded financial category, the ML system will simply learn and replicate that pervasive error. It is recommended that a gold-coded SIC sample is generated to train the first model. Then, for both SIC and SOC, manually-coded samples that could not be matched by the tool could form gold-coded data that could be used to retrain the model.

8. Future developments recommendations

Our recommendations for the future use of this tool are in two main strands, methodologically and strategically.

Methodologically

Methodologically, the project will benefit from introducing some extra training data to strengthen the model on under-represented classes. Recent AI developments are full of algorithms that can outperform logistic regression on this task, such as convolutional neural networks (CNN). CNNs have proven robust and efficient in text classification problems and it would be a great approach given the wealth of training data available.

We are aware from trials that just slightly more complex models like Random Forests can improve performance beyond the baseline Logistic Regression (LR) results we have presented. Finally, the addition of features such as BERT, a way of representing the meaning of words as multi-dimensional vectors, could also improve performance on all metrics.

Regarding new codes alerts or detection, which was a question we have been repeatedly asked, we believe that this could come as a separate project using established outlier detection techniques. In deployment of the current model, it should be the case that new professions will revert to manual coding – these samples should get low probability scores and fail to pass the match threshold.

Strategically

Strategically, in the short-term, the tool could be used in its current state, as it is proven to generalise well, however, an evaluation similar to the ones we conducted in this report on new datasets would be recommended. In the long-term, the Standard Industrial Classifications (SIC) and Standard Occupational Classifications (SOC) problem could be unified, by asking survey designers to ask similar questions to the ones we train our highest-performing models like census and Labour Force Survey (LFS)), this way we will need fewer models in the future and of course models will tend to become more accurate, confident and cost-effective, needing less manual labelling and less technical.

9. Concluding comments

During this project there was a lot of debate as to whether keeping a detailed knowledge base is unavoidable. One of the reasons being that if we rely on models and stop maintaining a knowledge base, we may lose the human expertise. However, the model could generate knowledge bases, by shortlisting the terms that correspond to the highest coefficients (weights) on the Logistic Regression model. This could be examined by experts and refined if necessary – both to quality assure the model and to refine the knowledge bases. This way a good knowledge base could be maintained with minimum cost and effort, while enjoying the advantages that a data-driven model could offer.

There is strong demand for a centralised solution, which would not only be more efficient but also improve consistency in coding to SIC and SOC. This would improve the quality of data and ultimately the quality of industry and occupation statistics.

Therefore, the Office for National Statistics (ONS)  is undertaking a further research project to compare and evaluate the rules-based coding tool with the ML approach described in this report. The objective of this research is to inform a recommendation to develop a SIC and SOC matching tool. A detailed evaluation of performance metrics on match rate and quality across the classification frameworks will be conducted, to understand where the methodologies perform well, and where they do not. An evaluation of data requirements will also be undertaken to understand the cost and sustainability of particular coding solutions, for instance, through continuous maintenance of multiple knowledge bases, versus gold coding of data. These results will be viewed alongside user requirements, to ensure any recommendation can be implemented across the range of statistical products that exist within the ONS.

The code for this project can be found on our private cloud’s (DAP) gitlab account. Most of the code used for this project is also available on our public github repo ace2, which has convenient modules for data and feature engineering for text classification purposes.

10. Contributors

With contributions from:

Evie Brown

Harriet Sands

Lucie Ferguson