In several studies, it has been suggested that task-switching performance is linked to processing speed. Here we argue that the relation between processing speed and high-level cognitive ability found in previous studies may be due to confounded measurements of processing speed and task-switching ability. In the present study, we required participants to complete an inspection time (IT) task to probe their processing speed. We employed conventional task-switching paradigms but applied a linear integrated speed-accuracy score (LISAS) which combines latency and accuracy scores to express task-switching ability. The results of regression analyses show that IT predicted average performance in task-switching paradigms. However, IT did not relate to any specific effects common in the task-switching task, which contradicts previous results. Our results suggest independent mechanisms of processing speed and tasks that require a high level of cognitive flexibility and control.