Feature Selection and Verification in PIAAC Problem-Solving Tests

The feature selection and verification processes in the context of real-world problem-solving tasks are essential for understanding how individuals approach and solve challenges. In this article, we will explore the insights gleaned from two specific problem sets from the Programme for the International Assessment of Adult Competencies (PIAAC): ‘Party Invitation’ and ‘Club Membership’.

Party Invitation

The ‘Party Invitation’ problem set involves the task of organizing emails into designated folders based on their purpose. This challenge not only tests reading comprehension but also evaluates users’ familiarity with their digital environment.

Feature Selection—Word2Vec Results

The analysis of participant actions segregated into performance groups—those scoring 0 versus those scoring 3—reveals significant distinctions in their action sequences. Data visualizations (Figures 3 and 4) illustrate these differences through vectorized representations.

Vector Space Analysis

For the score 0 group, the vector space is organized into three distinctive clusters:

• Cluster 1: Displays three mail-moving actions (MAIL_DROP, MAIL_MOVED, MAIL_DRAG). Notably, these actions are positioned far apart from other actions within this group’s vector space, contrasting with the score 3 group where these actions show coherent clustering with neighboring tasks.

• Cluster 2: While both groups exhibit differences in this cluster, the score 0 group presents tightly grouped actions without sub-clusters. In contrast, the score 3 group’s actions are more dispersed, suggesting a greater breadth in strategy and execution.

• Cluster 3: Interestingly, actions in this cluster do not reveal significant variances between the two groups, indicating consistent types of actions performed by both.

The mail-moving actions emerge as crucial differentiators, especially in the score 3 group, illustrated by a complex sequence of actions that demonstrates a rich context of usage.

Feature Verification—Doc2Vec, NN, and RF Results

To validate the significance of the mail-moving actions, three case studies were generated:

1. Case 1: Original sequences.
2. Case 2: Sequences limited to mail-moving actions.
3. Case 3: Sequences devoid of mail-moving actions.

The results from the Doc2Vec analysis show that:

• Case 1 achieves a silhouette score of 0.36.
• Case 2 demonstrates superior separation with a score of 0.491.
• Case 3 falls to a score of 0.216.

These observations underscore the potency of focusing on specific action types.

The neural network (NN) results reinforce this notion, showcasing predictive efficiencies between 0.850 and 0.962, with Case 2 submissions yielding the highest accuracies.

Analysis of Silhouette Scores

The silhouette scores may be perceived as modest; however, they can reflect the challenges posed by dimensionality reduction techniques and the varying densities within clusters. Despite these limitations, the clear differences in case 2’s clustering depict a coherent and interpretable structure.

Club Membership

In the ‘Club Membership’ problem set, participants must locate a membership ID number corresponding to a name and send it via email. This task evaluates their capability to navigate lookup systems and mailing functionalities effectively.

Feature Selection—Word2Vec Results

Visualizations of participant actions (Figures 6 and 7) reveal compelling comparisons between those who succeeded and those who faltered in the task. Here, the incorrect group’s actions appear divided into three clusters, with noteworthy discrepancies in processing sorting mechanics and environmental interactions.

• Cluster 1 highlights significant differences in sorting actions, which are underrepresented among both groups, hence not selected for verification.

• Cluster 2, on the other hand, emerges as a key area of interest, containing the mail-sending and environmental actions, pivotal for correctly executing the task.

Feature Verification—Doc2Vec, NN, and RF Results

Similar to the previous task, the three case scenarios highlighted above were again applicable:

• In Case 1, all actions were considered.
• Case 2 focused specifically on mail-sending actions.
• Case 3 excluded these actions entirely.

A descriptive analysis of the silhouette scores shows that Case 2 provides the clearest delineation of participant performance, confirming the importance of mail-sending behaviors in successful task completion.

Neural Network Performance

The efficient performance metrics displayed by the NN in this context (scores ranging from 0.862 to 0.917) indicate the reliability of mail-sending actions as markers of success. For Case 2, despite the limited action scope, the results remained robust, reflecting the cohesive relationship between mail-sending actions and their impact on task performance.

Conclusion

The insights from the ‘Party Invitation’ and ‘Club Membership’ problem sets demonstrate that specific actions—whether mail-moving or mail-sending—are not merely incidental but rather central to task success. The methodologies employed in feature selection and verification reveal the complex ways individuals interact with digital tools, highlighting the significant potential for predictive analytics in assessing problem-solving skills in adult competencies.